.. _sec_scheduler:
Öğrenme Oranını Zamanlama
=========================
Şimdiye kadar öncelikle ağırlık vektörlerinin güncellendikleri *oran*
yerine ağırlık vektörlerinin nasıl güncelleneceğine dair optimizasyon
*algoritmalarına* odaklandık. Bununla birlikte, öğrenme oranının
ayarlanması genellikle gerçek algoritma kadar önemlidir. Göz önünde
bulundurulması gereken bir dizi husus vardır:
- En çok açıkçası öğrenme oranının *büyüklüğü* önemlidir. Çok büyükse,
optimizasyon ıraksar, eğer çok küçükse, eğitilmesi çok uzun sürer
veya yarı-optimal bir sonuç elde ederiz. Daha önce problemin
sağlamlık sayısının önemli olduğunu gördük (ayrıntılar için bkz.
:numref:`sec_momentum`). Sezgisel olarak, en hassas yöndeki değişim
miktarının en az hassas olanına oranıdır.
- İkincisi, sönme oranı da aynı derecede önemlidir. Öğrenme oranı büyük
kalırsa, en küçük seviyeyi atlayabilir ve böylece eniyiye
ulaşamayabiliriz. :numref:`sec_minibatch_sgd` içinde bunu ayrıntılı
olarak tartıştık ve :numref:`sec_sgd` içinde performans
garantilerini analiz ettik. Kısacası, sönme oranını istiyoruz, ama
muhtemelen dışbükey problemler için
:math:`\mathcal{O}(t^{-\frac{1}{2}})` daha yavaş olanlar iyi bir
seçim olurdu.
- Eşit derecede önemli olan bir diğer yön ise *ilklemedir*. Bu, hem
parametrelerin başlangıçta nasıl ayarlandığı (ayrıntılar için
bkz.:numref:\ ``sec_numerical_stability``) hem de başlangıçta nasıl
evrimleştikleri de ilgilidir. Bu, *ısınma* diye tabir edilir, yani
başlangıçta çözüme doğru ne kadar hızlı ilerlemeye başladığımızla
ilgilidir. Başlangıçta büyük adımlar yararlı olmayabilir, özellikle
de ilk parametre kümesi rastgele olduğundan. İlk güncelleme
yönergeleri de oldukça anlamsız olabilir.
- Son olarak, döngüsel öğrenme hızı ayarlaması gerçekleştiren bir dizi
optimizasyon türü vardır. Bu, bu bölümün kapsamı dışındadır.
Okuyucuya :cite:`Izmailov.Podoprikhin.Garipov.ea.2018`
çalışmasındaki ayrıntıları gözden geçirmesini tavsiye ederiz,
örneğin, parametrelerin izlediği *yolun* üzerinden ortalama alarak
daha iyi çözümler elde etmek gibi.
Öğrenme oranlarını yönetmek için gereken çok fazla ayrıntı olduğu göz
önüne alındığında, çoğu derin öğrenme çerçevesinin bununla otomatik
olarak başa çıkabilmesi için araçlar vardır. Bu bölümde, farklı
zamanlamaların doğruluk üzerindeki etkilerini gözden geçireceğiz ve
ayrıca bunun bir *öğrenme oranı zamanlayıcısı* aracılığıyla nasıl
verimli bir şekilde yönetilebileceğini göstereceğiz.
Basit Örnek Problem
-------------------
Kolayca hesaplanabilecek kadar ucuz, ancak bazı önemli hususları
göstermek için yeterince açık olmayan bir basit örnek problem ile
başlıyoruz. Bunun için Fashion-MNIST'e uygulandığı gibi LeNet'in biraz
modern bir sürümünü seçin (``relu`` yerine ``sigmoid`` aktivasyon,
MaxPooling (Maksimum Havuzlama) yerine AveragePooling (Ortalama
Havuzlama)). Dahası, ağı performans için melezleştiriyoruz. Kodun çoğu
standart olduğundan, daha fazla ayrıntılı tartışma yapmadan sadece temel
bilgileri sunuyoruz. Gerektiğinde anımsamak için bkz.
:numref:`chap_cnn`.
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
%matplotlib inline
from d2l import mxnet as d2l
from mxnet import autograd, gluon, init, lr_scheduler, np, npx
from mxnet.gluon import nn
npx.set_np()
net = nn.HybridSequential()
net.add(nn.Conv2D(channels=6, kernel_size=5, padding=2, activation='relu'),
nn.MaxPool2D(pool_size=2, strides=2),
nn.Conv2D(channels=16, kernel_size=5, activation='relu'),
nn.MaxPool2D(pool_size=2, strides=2),
nn.Dense(120, activation='relu'),
nn.Dense(84, activation='relu'),
nn.Dense(10))
net.hybridize()
loss = gluon.loss.SoftmaxCrossEntropyLoss()
device = d2l.try_gpu()
batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)
# Kod, evrişimli sinir ağları bölümünün lenet kısmında tanımlanan
# `d2l.train_ch6` ile neredeyse aynıdır.
def train(net, train_iter, test_iter, num_epochs, loss, trainer, device):
net.initialize(force_reinit=True, ctx=device, init=init.Xavier())
animator = d2l.Animator(xlabel='epoch', xlim=[0, num_epochs],
legend=['train loss', 'train acc', 'test acc'])
for epoch in range(num_epochs):
metric = d2l.Accumulator(3) # train_loss, train_acc, num_examples
for i, (X, y) in enumerate(train_iter):
X, y = X.as_in_ctx(device), y.as_in_ctx(device)
with autograd.record():
y_hat = net(X)
l = loss(y_hat, y)
l.backward()
trainer.step(X.shape[0])
metric.add(l.sum(), d2l.accuracy(y_hat, y), X.shape[0])
train_loss = metric[0] / metric[2]
train_acc = metric[1] / metric[2]
if (i + 1) % 50 == 0:
animator.add(epoch + i / len(train_iter),
(train_loss, train_acc, None))
test_acc = d2l.evaluate_accuracy_gpu(net, test_iter)
animator.add(epoch + 1, (None, None, test_acc))
print(f'train loss {train_loss:.3f}, train acc {train_acc:.3f}, '
f'test acc {test_acc:.3f}')
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
%matplotlib inline
import math
import torch
from torch import nn
from torch.optim import lr_scheduler
from d2l import torch as d2l
def net_fn():
model = nn.Sequential(
nn.Conv2d(1, 6, kernel_size=5, padding=2), nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),
nn.Conv2d(6, 16, kernel_size=5), nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),
nn.Flatten(),
nn.Linear(16 * 5 * 5, 120), nn.ReLU(),
nn.Linear(120, 84), nn.ReLU(),
nn.Linear(84, 10))
return model
loss = nn.CrossEntropyLoss()
device = d2l.try_gpu()
batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)
# Kod, evrişimli sinir ağları bölümünün lenet kısmında tanımlanan
# `d2l.train_ch6` ile neredeyse aynıdır.
def train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
scheduler=None):
net.to(device)
animator = d2l.Animator(xlabel='epoch', xlim=[0, num_epochs],
legend=['train loss', 'train acc', 'test acc'])
for epoch in range(num_epochs):
metric = d2l.Accumulator(3) # train_loss, train_acc, num_examples
for i, (X, y) in enumerate(train_iter):
net.train()
trainer.zero_grad()
X, y = X.to(device), y.to(device)
y_hat = net(X)
l = loss(y_hat, y)
l.backward()
trainer.step()
with torch.no_grad():
metric.add(l * X.shape[0], d2l.accuracy(y_hat, y), X.shape[0])
train_loss = metric[0] / metric[2]
train_acc = metric[1] / metric[2]
if (i + 1) % 50 == 0:
animator.add(epoch + i / len(train_iter),
(train_loss, train_acc, None))
test_acc = d2l.evaluate_accuracy_gpu(net, test_iter)
animator.add(epoch+1, (None, None, test_acc))
if scheduler:
if scheduler.__module__ == lr_scheduler.__name__:
# Using PyTorch In-Built scheduler
scheduler.step()
else:
# Using custom defined scheduler
for param_group in trainer.param_groups:
param_group['lr'] = scheduler(epoch)
print(f'train loss {train_loss:.3f}, train acc {train_acc:.3f}, '
f'test acc {test_acc:.3f}')
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
%matplotlib inline
import math
import tensorflow as tf
from d2l import tensorflow as d2l
from tensorflow.keras.callbacks import LearningRateScheduler
def net():
return tf.keras.models.Sequential([
tf.keras.layers.Conv2D(filters=6, kernel_size=5, activation='relu',
padding='same'),
tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
tf.keras.layers.Conv2D(filters=16, kernel_size=5,
activation='relu'),
tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(120, activation='relu'),
tf.keras.layers.Dense(84, activation='sigmoid'),
tf.keras.layers.Dense(10)])
batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)
# Kod, evrişimli sinir ağları bölümünün lenet kısmında tanımlanan
# `d2l.train_ch6` ile neredeyse aynıdır.
def train(net_fn, train_iter, test_iter, num_epochs, lr,
device=d2l.try_gpu(), custom_callback = False):
device_name = device._device_name
strategy = tf.distribute.OneDeviceStrategy(device_name)
with strategy.scope():
optimizer = tf.keras.optimizers.SGD(learning_rate=lr)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
net = net_fn()
net.compile(optimizer=optimizer, loss=loss, metrics=['accuracy'])
callback = d2l.TrainCallback(net, train_iter, test_iter, num_epochs,
device_name)
if custom_callback is False:
net.fit(train_iter, epochs=num_epochs, verbose=0,
callbacks=[callback])
else:
net.fit(train_iter, epochs=num_epochs, verbose=0,
callbacks=[callback, custom_callback])
return net
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
lr, num_epochs = 0.3, 30
net.initialize(force_reinit=True, ctx=device, init=init.Xavier())
trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': lr})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.171, train acc 0.935, test acc 0.874
.. figure:: output_lr-scheduler_1dfeb6_15_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
lr, num_epochs = 0.3, 30
net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=lr)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.181, train acc 0.930, test acc 0.892
.. figure:: output_lr-scheduler_1dfeb6_18_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
lr, num_epochs = 0.3, 30
train(net, train_iter, test_iter, num_epochs, lr)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
loss 0.223, train acc 0.917, test acc 0.886
61325.8 examples/sec on /GPU:0
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. figure:: output_lr-scheduler_1dfeb6_21_2.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
trainer.set_learning_rate(0.1)
print(f'learning rate is now {trainer.learning_rate:.2f}')
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
learning rate is now 0.10
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
lr = 0.1
trainer.param_groups[0]["lr"] = lr
print(f'learning rate is now {trainer.param_groups[0]["lr"]:.2f}')
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
learning rate is now 0.10
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
lr = 0.1
dummy_model = tf.keras.models.Sequential([tf.keras.layers.Dense(10)])
dummy_model.compile(tf.keras.optimizers.SGD(learning_rate=lr), loss='mse')
print(f'learning rate is now ,', dummy_model.optimizer.lr.numpy())
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
learning rate is now , 0.1
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
scheduler = SquareRootScheduler(lr=0.1)
d2l.plot(np.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_41_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
scheduler = SquareRootScheduler(lr=0.1)
d2l.plot(torch.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_44_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
scheduler = SquareRootScheduler(lr=0.1)
d2l.plot(tf.range(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_47_0.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
trainer = gluon.Trainer(net.collect_params(), 'sgd',
{'lr_scheduler': scheduler})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.522, train acc 0.812, test acc 0.815
.. figure:: output_lr-scheduler_1dfeb6_53_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
scheduler)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.283, train acc 0.897, test acc 0.873
.. figure:: output_lr-scheduler_1dfeb6_56_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
train(net, train_iter, test_iter, num_epochs, lr,
custom_callback=LearningRateScheduler(scheduler))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
loss 0.379, train acc 0.863, test acc 0.843
61240.3 examples/sec on /GPU:0
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. figure:: output_lr-scheduler_1dfeb6_59_2.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class FactorScheduler:
def __init__(self, factor=1, stop_factor_lr=1e-7, base_lr=0.1):
self.factor = factor
self.stop_factor_lr = stop_factor_lr
self.base_lr = base_lr
def __call__(self, num_update):
self.base_lr = max(self.stop_factor_lr, self.base_lr * self.factor)
return self.base_lr
scheduler = FactorScheduler(factor=0.9, stop_factor_lr=1e-2, base_lr=2.0)
d2l.plot(np.arange(50), [scheduler(t) for t in range(50)])
.. figure:: output_lr-scheduler_1dfeb6_65_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class FactorScheduler:
def __init__(self, factor=1, stop_factor_lr=1e-7, base_lr=0.1):
self.factor = factor
self.stop_factor_lr = stop_factor_lr
self.base_lr = base_lr
def __call__(self, num_update):
self.base_lr = max(self.stop_factor_lr, self.base_lr * self.factor)
return self.base_lr
scheduler = FactorScheduler(factor=0.9, stop_factor_lr=1e-2, base_lr=2.0)
d2l.plot(torch.arange(50), [scheduler(t) for t in range(50)])
.. figure:: output_lr-scheduler_1dfeb6_68_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class FactorScheduler:
def __init__(self, factor=1, stop_factor_lr=1e-7, base_lr=0.1):
self.factor = factor
self.stop_factor_lr = stop_factor_lr
self.base_lr = base_lr
def __call__(self, num_update):
self.base_lr = max(self.stop_factor_lr, self.base_lr * self.factor)
return self.base_lr
scheduler = FactorScheduler(factor=0.9, stop_factor_lr=1e-2, base_lr=2.0)
d2l.plot(tf.range(50), [scheduler(t) for t in range(50)])
.. figure:: output_lr-scheduler_1dfeb6_71_0.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
scheduler = lr_scheduler.MultiFactorScheduler(step=[15, 30], factor=0.5,
base_lr=0.5)
d2l.plot(np.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_77_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=0.5)
scheduler = lr_scheduler.MultiStepLR(trainer, milestones=[15, 30], gamma=0.5)
def get_lr(trainer, scheduler):
lr = scheduler.get_last_lr()[0]
trainer.step()
scheduler.step()
return lr
d2l.plot(torch.arange(num_epochs), [get_lr(trainer, scheduler)
for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_80_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class MultiFactorScheduler:
def __init__(self, step, factor, base_lr):
self.step = step
self.factor = factor
self.base_lr = base_lr
def __call__(self, epoch):
if epoch in self.step:
self.base_lr = self.base_lr * self.factor
return self.base_lr
else:
return self.base_lr
scheduler = MultiFactorScheduler(step=[15, 30], factor=0.5, base_lr=0.5)
d2l.plot(tf.range(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_83_0.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
trainer = gluon.Trainer(net.collect_params(), 'sgd',
{'lr_scheduler': scheduler})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.177, train acc 0.933, test acc 0.901
.. figure:: output_lr-scheduler_1dfeb6_89_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
scheduler)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.204, train acc 0.922, test acc 0.888
.. figure:: output_lr-scheduler_1dfeb6_92_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
train(net, train_iter, test_iter, num_epochs, lr,
custom_callback=LearningRateScheduler(scheduler))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
loss 0.228, train acc 0.915, test acc 0.890
60173.8 examples/sec on /GPU:0
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. figure:: output_lr-scheduler_1dfeb6_95_2.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
scheduler = lr_scheduler.CosineScheduler(max_update=20, base_lr=0.3,
final_lr=0.01)
d2l.plot(np.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_101_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class CosineScheduler:
def __init__(self, max_update, base_lr=0.01, final_lr=0,
warmup_steps=0, warmup_begin_lr=0):
self.base_lr_orig = base_lr
self.max_update = max_update
self.final_lr = final_lr
self.warmup_steps = warmup_steps
self.warmup_begin_lr = warmup_begin_lr
self.max_steps = self.max_update - self.warmup_steps
def get_warmup_lr(self, epoch):
increase = (self.base_lr_orig - self.warmup_begin_lr) \
* float(epoch) / float(self.warmup_steps)
return self.warmup_begin_lr + increase
def __call__(self, epoch):
if epoch < self.warmup_steps:
return self.get_warmup_lr(epoch)
if epoch <= self.max_update:
self.base_lr = self.final_lr + (
self.base_lr_orig - self.final_lr) * (1 + math.cos(
math.pi * (epoch - self.warmup_steps) / self.max_steps)) / 2
return self.base_lr
scheduler = CosineScheduler(max_update=20, base_lr=0.3, final_lr=0.01)
d2l.plot(torch.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_104_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
class CosineScheduler:
def __init__(self, max_update, base_lr=0.01, final_lr=0,
warmup_steps=0, warmup_begin_lr=0):
self.base_lr_orig = base_lr
self.max_update = max_update
self.final_lr = final_lr
self.warmup_steps = warmup_steps
self.warmup_begin_lr = warmup_begin_lr
self.max_steps = self.max_update - self.warmup_steps
def get_warmup_lr(self, epoch):
increase = (self.base_lr_orig - self.warmup_begin_lr) \
* float(epoch) / float(self.warmup_steps)
return self.warmup_begin_lr + increase
def __call__(self, epoch):
if epoch < self.warmup_steps:
return self.get_warmup_lr(epoch)
if epoch <= self.max_update:
self.base_lr = self.final_lr + (
self.base_lr_orig - self.final_lr) * (1 + math.cos(
math.pi * (epoch - self.warmup_steps) / self.max_steps)) / 2
return self.base_lr
scheduler = CosineScheduler(max_update=20, base_lr=0.3, final_lr=0.01)
d2l.plot(tf.range(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_107_0.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
trainer = gluon.Trainer(net.collect_params(), 'sgd',
{'lr_scheduler': scheduler})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.345, train acc 0.878, test acc 0.873
.. figure:: output_lr-scheduler_1dfeb6_113_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=0.3)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
scheduler)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.178, train acc 0.934, test acc 0.899
.. figure:: output_lr-scheduler_1dfeb6_116_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
train(net, train_iter, test_iter, num_epochs, lr,
custom_callback=LearningRateScheduler(scheduler))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
loss 0.261, train acc 0.904, test acc 0.880
60457.0 examples/sec on /GPU:0
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. figure:: output_lr-scheduler_1dfeb6_119_2.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
scheduler = lr_scheduler.CosineScheduler(20, warmup_steps=5, base_lr=0.3,
final_lr=0.01)
d2l.plot(np.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_125_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
scheduler = CosineScheduler(20, warmup_steps=5, base_lr=0.3, final_lr=0.01)
d2l.plot(torch.arange(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_128_0.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
scheduler = CosineScheduler(20, warmup_steps=5, base_lr=0.3, final_lr=0.01)
d2l.plot(tf.range(num_epochs), [scheduler(t) for t in range(num_epochs)])
.. figure:: output_lr-scheduler_1dfeb6_131_0.svg
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
trainer = gluon.Trainer(net.collect_params(), 'sgd',
{'lr_scheduler': scheduler})
train(net, train_iter, test_iter, num_epochs, loss, trainer, device)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.347, train acc 0.874, test acc 0.873
.. figure:: output_lr-scheduler_1dfeb6_137_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net = net_fn()
trainer = torch.optim.SGD(net.parameters(), lr=0.3)
train(net, train_iter, test_iter, num_epochs, loss, trainer, device,
scheduler)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
train loss 0.177, train acc 0.934, test acc 0.907
.. figure:: output_lr-scheduler_1dfeb6_140_1.svg
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
train(net, train_iter, test_iter, num_epochs, lr,
custom_callback=LearningRateScheduler(scheduler))
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
loss 0.272, train acc 0.901, test acc 0.881
62052.2 examples/sec on /GPU:0
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
.. figure:: output_lr-scheduler_1dfeb6_143_2.svg
.. raw:: html
.. raw:: html
.. raw:: html
`Tartışmalar `__
.. raw:: html
.. raw:: html
`Tartışmalar `__
.. raw:: html
.. raw:: html
`Tartışmalar `__
.. raw:: html
.. raw:: html