Shortcuts

Hyperparameter tuning with Ray Tune

Hyperparameter tuning can make the difference between an average model and a highly accurate one. Often simple things like choosing a different learning rate or changing a network layer size can have a dramatic impact on your model performance.

Fortunately, there are tools that help with finding the best combination of parameters. Ray Tune is an industry standard tool for distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search algorithms, integrates with TensorBoard and other analysis libraries, and natively supports distributed training through Ray’s distributed machine learning engine.

In this tutorial, we will show you how to integrate Ray Tune into your PyTorch training workflow. We will extend this tutorial from the PyTorch documentation for training a CIFAR10 image classifier.

As you will see, we only need to add some slight modifications. In particular, we need to

  1. wrap data loading and training in functions,
  2. make some network parameters configurable,
  3. add checkpointing (optional),
  4. and define the search space for the model tuning

To run this tutorial, please make sure the following packages are installed:

  • ray[tune]: Distributed hyperparameter tuning library
  • torchvision: For the data transformers

Setup / Imports

Let’s start with the imports:

from functools import partial
import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import random_split
import torchvision
import torchvision.transforms as transforms
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler

Most of the imports are needed for building the PyTorch model. Only the last three imports are for Ray Tune.

Data loaders

We wrap the data loaders in their own function and pass a global data directory. This way we can share a data directory between different trials.

def load_data(data_dir="./data"):
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

    trainset = torchvision.datasets.CIFAR10(
        root=data_dir, train=True, download=True, transform=transform)

    testset = torchvision.datasets.CIFAR10(
        root=data_dir, train=False, download=True, transform=transform)

    return trainset, testset

Configurable neural network

We can only tune those parameters that are configurable. In this example, we can specify the layer sizes of the fully connected layers:

class Net(nn.Module):
    def __init__(self, l1=120, l2=84):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, l1)
        self.fc2 = nn.Linear(l1, l2)
        self.fc3 = nn.Linear(l2, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The train function

Now it gets interesting, because we introduce some changes to the example from the PyTorch documentation.

We wrap the training script in a function train_cifar(config, checkpoint_dir=None, data_dir=None). As you can guess, the config parameter will receive the hyperparameters we would like to train with. The checkpoint_dir parameter is used to restore checkpoints. The data_dir specifies the directory where we load and store the data, so multiple runs can share the same data source.

net = Net(config["l1"], config["l2"])

if checkpoint_dir:
    model_state, optimizer_state = torch.load(
        os.path.join(checkpoint_dir, "checkpoint"))
    net.load_state_dict(model_state)
    optimizer.load_state_dict(optimizer_state)

The learning rate of the optimizer is made configurable, too:

optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

We also split the training data into a training and validation subset. We thus train on 80% of the data and calculate the validation loss on the remaining 20%. The batch sizes with which we iterate through the training and test sets are configurable as well.

Adding (multi) GPU support with DataParallel

Image classification benefits largely from GPUs. Luckily, we can continue to use PyTorch’s abstractions in Ray Tune. Thus, we can wrap our model in nn.DataParallel to support data parallel training on multiple GPUs:

device = "cpu"
if torch.cuda.is_available():
    device = "cuda:0"
    if torch.cuda.device_count() > 1:
        net = nn.DataParallel(net)
net.to(device)

By using a device variable we make sure that training also works when we have no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly, like this:

for i, data in enumerate(trainloader, 0):
    inputs, labels = data
    inputs, labels = inputs.to(device), labels.to(device)

The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray also supports fractional GPUs so we can share GPUs among trials, as long as the model still fits on the GPU memory. We’ll come back to that later.

Communicating with Ray Tune

The most interesting part is the communication with Ray Tune:

with tune.checkpoint_dir(epoch) as checkpoint_dir:
    path = os.path.join(checkpoint_dir, "checkpoint")
    torch.save((net.state_dict(), optimizer.state_dict()), path)

tune.report(loss=(val_loss / val_steps), accuracy=correct / total)

Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically, we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics to decide which hyperparameter configuration lead to the best results. These metrics can also be used to stop bad performing trials early in order to avoid wasting resources on those trials.

The checkpoint saving is optional, however, it is necessary if we wanted to use advanced schedulers like Population Based Training. Also, by saving the checkpoint we can later load the trained models and validate them on a test set.

Full training function

The full code example looks like this:

def train_cifar(config, checkpoint_dir=None, data_dir=None):
    net = Net(config["l1"], config["l2"])

    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if torch.cuda.device_count() > 1:
            net = nn.DataParallel(net)
    net.to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

    if checkpoint_dir:
        model_state, optimizer_state = torch.load(
            os.path.join(checkpoint_dir, "checkpoint"))
        net.load_state_dict(model_state)
        optimizer.load_state_dict(optimizer_state)

    trainset, testset = load_data(data_dir)

    test_abs = int(len(trainset) * 0.8)
    train_subset, val_subset = random_split(
        trainset, [test_abs, len(trainset) - test_abs])

    trainloader = torch.utils.data.DataLoader(
        train_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)
    valloader = torch.utils.data.DataLoader(
        val_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)

    for epoch in range(10):  # loop over the dataset multiple times
        running_loss = 0.0
        epoch_steps = 0
        for i, data in enumerate(trainloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            epoch_steps += 1
            if i % 2000 == 1999:  # print every 2000 mini-batches
                print("[%d, %5d] loss: %.3f" % (epoch + 1, i + 1,
                                                running_loss / epoch_steps))
                running_loss = 0.0

        # Validation loss
        val_loss = 0.0
        val_steps = 0
        total = 0
        correct = 0
        for i, data in enumerate(valloader, 0):
            with torch.no_grad():
                inputs, labels = data
                inputs, labels = inputs.to(device), labels.to(device)

                outputs = net(inputs)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()

                loss = criterion(outputs, labels)
                val_loss += loss.cpu().numpy()
                val_steps += 1

        with tune.checkpoint_dir(epoch) as checkpoint_dir:
            path = os.path.join(checkpoint_dir, "checkpoint")
            torch.save((net.state_dict(), optimizer.state_dict()), path)

        tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
    print("Finished Training")

As you can see, most of the code is adapted directly from the original example.

Test set accuracy

Commonly the performance of a machine learning model is tested on a hold-out test set with data that has not been used for training the model. We also wrap this in a function:

def test_accuracy(net, device="cpu"):
    trainset, testset = load_data()

    testloader = torch.utils.data.DataLoader(
        testset, batch_size=4, shuffle=False, num_workers=2)

    correct = 0
    total = 0
    with torch.no_grad():
        for data in testloader:
            images, labels = data
            images, labels = images.to(device), labels.to(device)
            outputs = net(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    return correct / total

The function also expects a device parameter, so we can do the test set validation on a GPU.

Configuring the search space

Lastly, we need to define Ray Tune’s search space. Here is an example:

config = {
    "l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "lr": tune.loguniform(1e-4, 1e-1),
    "batch_size": tune.choice([2, 4, 8, 16])
}

The tune.sample_from() function makes it possible to define your own sample methods to obtain hyperparameters. In this example, the l1 and l2 parameters should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256. The lr (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly, the batch size is a choice between 2, 4, 8, and 16.

At each trial, Ray Tune will now randomly sample a combination of parameters from these search spaces. It will then train a number of models in parallel and find the best performing one among these. We also use the ASHAScheduler which will terminate bad performing trials early.

We wrap the train_cifar function with functools.partial to set the constant data_dir parameter. We can also tell Ray Tune what resources should be available for each trial:

gpus_per_trial = 2
# ...
result = tune.run(
    partial(train_cifar, data_dir=data_dir),
    resources_per_trial={"cpu": 8, "gpu": gpus_per_trial},
    config=config,
    num_samples=num_samples,
    scheduler=scheduler,
    progress_reporter=reporter,
    checkpoint_at_end=True)

You can specify the number of CPUs, which are then available e.g. to increase the num_workers of the PyTorch DataLoader instances. The selected number of GPUs are made visible to PyTorch in each trial. Trials do not have access to GPUs that haven’t been requested for them - so you don’t have to care about two trials using the same set of resources.

Here we can also specify fractional GPUs, so something like gpus_per_trial=0.5 is completely valid. The trials will then share GPUs among each other. You just have to make sure that the models still fit in the GPU memory.

After training the models, we will find the best performing one and load the trained network from the checkpoint file. We then obtain the test set accuracy and report everything by printing.

The full main function looks like this:

def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):
    data_dir = os.path.abspath("./data")
    load_data(data_dir)
    config = {
        "l1": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "l2": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "lr": tune.loguniform(1e-4, 1e-1),
        "batch_size": tune.choice([2, 4, 8, 16])
    }
    scheduler = ASHAScheduler(
        metric="loss",
        mode="min",
        max_t=max_num_epochs,
        grace_period=1,
        reduction_factor=2)
    reporter = CLIReporter(
        # parameter_columns=["l1", "l2", "lr", "batch_size"],
        metric_columns=["loss", "accuracy", "training_iteration"])
    result = tune.run(
        partial(train_cifar, data_dir=data_dir),
        resources_per_trial={"cpu": 2, "gpu": gpus_per_trial},
        config=config,
        num_samples=num_samples,
        scheduler=scheduler,
        progress_reporter=reporter)

    best_trial = result.get_best_trial("loss", "min", "last")
    print("Best trial config: {}".format(best_trial.config))
    print("Best trial final validation loss: {}".format(
        best_trial.last_result["loss"]))
    print("Best trial final validation accuracy: {}".format(
        best_trial.last_result["accuracy"]))

    best_trained_model = Net(best_trial.config["l1"], best_trial.config["l2"])
    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if gpus_per_trial > 1:
            best_trained_model = nn.DataParallel(best_trained_model)
    best_trained_model.to(device)

    best_checkpoint_dir = best_trial.checkpoint.value
    model_state, optimizer_state = torch.load(os.path.join(
        best_checkpoint_dir, "checkpoint"))
    best_trained_model.load_state_dict(model_state)

    test_acc = test_accuracy(best_trained_model, device)
    print("Best trial test set accuracy: {}".format(test_acc))


if __name__ == "__main__":
    # You can change the number of GPUs per trial here:
    main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)

Out:

Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /workspace/ko-latest/beginner_source/data/cifar-10-python.tar.gz
Extracting /workspace/ko-latest/beginner_source/data/cifar-10-python.tar.gz to /workspace/ko-latest/beginner_source/data
Files already downloaded and verified
== Status ==
Memory usage on this node: 14.3/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (9 PENDING, 1 RUNNING)
+---------------------+----------+-------+--------------+------+------+-------------+
| Trial name          | status   | loc   |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-------+--------------+------+------+-------------|
| DEFAULT_f201f_00000 | RUNNING  |       |           16 |  128 |   16 | 0.0173062   |
| DEFAULT_f201f_00001 | PENDING  |       |            2 |  128 |    8 | 0.000447305 |
| DEFAULT_f201f_00002 | PENDING  |       |            2 |  256 |  256 | 0.0119053   |
| DEFAULT_f201f_00003 | PENDING  |       |            2 |   32 |   16 | 0.00404483  |
| DEFAULT_f201f_00004 | PENDING  |       |            4 |    8 |   16 | 0.00134526  |
| DEFAULT_f201f_00005 | PENDING  |       |            2 |   16 |   16 | 0.00126487  |
| DEFAULT_f201f_00006 | PENDING  |       |            2 |  256 |   16 | 0.0404286   |
| DEFAULT_f201f_00007 | PENDING  |       |            8 |   64 |  256 | 0.000285913 |
| DEFAULT_f201f_00008 | PENDING  |       |            8 |   32 |   16 | 0.00172471  |
| DEFAULT_f201f_00009 | PENDING  |       |            8 |   64 |   16 | 0.0574224   |
+---------------------+----------+-------+--------------+------+------+-------------+


(pid=5085) Files already downloaded and verified
(pid=5088) Files already downloaded and verified
(pid=5090) Files already downloaded and verified
(pid=5086) Files already downloaded and verified
(pid=5085) Files already downloaded and verified
(pid=5088) Files already downloaded and verified
(pid=5090) Files already downloaded and verified
(pid=5086) Files already downloaded and verified
== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-------+--------------+------+------+-------------+
| Trial name          | status   | loc   |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-------+--------------+------+------+-------------|
| DEFAULT_f201f_00000 | RUNNING  |       |           16 |  128 |   16 | 0.0173062   |
| DEFAULT_f201f_00001 | RUNNING  |       |            2 |  128 |    8 | 0.000447305 |
| DEFAULT_f201f_00002 | RUNNING  |       |            2 |  256 |  256 | 0.0119053   |
| DEFAULT_f201f_00003 | RUNNING  |       |            2 |   32 |   16 | 0.00404483  |
| DEFAULT_f201f_00004 | PENDING  |       |            4 |    8 |   16 | 0.00134526  |
| DEFAULT_f201f_00005 | PENDING  |       |            2 |   16 |   16 | 0.00126487  |
| DEFAULT_f201f_00006 | PENDING  |       |            2 |  256 |   16 | 0.0404286   |
| DEFAULT_f201f_00007 | PENDING  |       |            8 |   64 |  256 | 0.000285913 |
| DEFAULT_f201f_00008 | PENDING  |       |            8 |   32 |   16 | 0.00172471  |
| DEFAULT_f201f_00009 | PENDING  |       |            8 |   64 |   16 | 0.0574224   |
+---------------------+----------+-------+--------------+------+------+-------------+


(pid=5086) [1,  2000] loss: 2.144
(pid=5090) [1,  2000] loss: 2.244
(pid=5088) [1,  2000] loss: 2.299
(pid=5085) [1,  2000] loss: 1.794
(pid=5086) [1,  4000] loss: 1.006
(pid=5088) [1,  4000] loss: 1.054
(pid=5090) [1,  4000] loss: 1.110
Result for DEFAULT_f201f_00000:
  accuracy: 0.4215
  date: 2021-07-29_14-12-50
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 1.6012202549934387
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 46.481757164001465
  time_this_iter_s: 46.481757164001465
  time_total_s: 46.481757164001465
  timestamp: 1627567970
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 15.8/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING  | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.60122 |     0.4215 |                    1 |
| DEFAULT_f201f_00001 | RUNNING  |                 |            2 |  128 |    8 | 0.000447305 |         |            |                      |
| DEFAULT_f201f_00002 | RUNNING  |                 |            2 |  256 |  256 | 0.0119053   |         |            |                      |
| DEFAULT_f201f_00003 | RUNNING  |                 |            2 |   32 |   16 | 0.00404483  |         |            |                      |
| DEFAULT_f201f_00004 | PENDING  |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING  |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING  |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING  |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING  |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING  |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5086) [1,  6000] loss: 0.648
(pid=5088) [1,  6000] loss: 0.641
(pid=5086) [1,  8000] loss: 0.499
(pid=5090) [1,  6000] loss: 0.720
(pid=5088) [1,  8000] loss: 0.448
(pid=5085) [2,  2000] loss: 1.567
(pid=5088) [1, 10000] loss: 0.347
(pid=5086) [1, 10000] loss: 0.386
Result for DEFAULT_f201f_00000:
  accuracy: 0.4254
  date: 2021-07-29_14-13-35
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 2
  loss: 1.6653225509643554
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 91.32741332054138
  time_this_iter_s: 44.84565615653992
  time_total_s: 91.32741332054138
  timestamp: 1627568015
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING  | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.66532 |     0.4254 |                    2 |
| DEFAULT_f201f_00001 | RUNNING  |                 |            2 |  128 |    8 | 0.000447305 |         |            |                      |
| DEFAULT_f201f_00002 | RUNNING  |                 |            2 |  256 |  256 | 0.0119053   |         |            |                      |
| DEFAULT_f201f_00003 | RUNNING  |                 |            2 |   32 |   16 | 0.00404483  |         |            |                      |
| DEFAULT_f201f_00004 | PENDING  |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING  |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING  |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING  |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING  |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING  |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5090) [1,  8000] loss: 0.539
(pid=5086) [1, 12000] loss: 0.321
(pid=5088) [1, 12000] loss: 0.279
(pid=5085) [3,  2000] loss: 1.525
(pid=5086) [1, 14000] loss: 0.276
(pid=5090) [1, 10000] loss: 0.427
(pid=5088) [1, 14000] loss: 0.234
Result for DEFAULT_f201f_00000:
  accuracy: 0.4648
  date: 2021-07-29_14-14-16
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 3
  loss: 1.5220761643409728
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 132.20206356048584
  time_this_iter_s: 40.87465023994446
  time_total_s: 132.20206356048584
  timestamp: 1627568056
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING  | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.52208 |     0.4648 |                    3 |
| DEFAULT_f201f_00001 | RUNNING  |                 |            2 |  128 |    8 | 0.000447305 |         |            |                      |
| DEFAULT_f201f_00002 | RUNNING  |                 |            2 |  256 |  256 | 0.0119053   |         |            |                      |
| DEFAULT_f201f_00003 | RUNNING  |                 |            2 |   32 |   16 | 0.00404483  |         |            |                      |
| DEFAULT_f201f_00004 | PENDING  |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING  |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING  |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING  |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING  |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING  |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5086) [1, 16000] loss: 0.244
(pid=5090) [1, 12000] loss: 0.357
(pid=5088) [1, 16000] loss: 0.202
(pid=5086) [1, 18000] loss: 0.211
(pid=5085) [4,  2000] loss: 1.498
(pid=5088) [1, 18000] loss: 0.176
Result for DEFAULT_f201f_00000:
  accuracy: 0.4372
  date: 2021-07-29_14-14-53
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 4
  loss: 1.6066401032447815
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 168.9676070213318
  time_this_iter_s: 36.76554346084595
  time_total_s: 168.9676070213318
  timestamp: 1627568093
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING  | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.60664 |     0.4372 |                    4 |
| DEFAULT_f201f_00001 | RUNNING  |                 |            2 |  128 |    8 | 0.000447305 |         |            |                      |
| DEFAULT_f201f_00002 | RUNNING  |                 |            2 |  256 |  256 | 0.0119053   |         |            |                      |
| DEFAULT_f201f_00003 | RUNNING  |                 |            2 |   32 |   16 | 0.00404483  |         |            |                      |
| DEFAULT_f201f_00004 | PENDING  |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING  |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING  |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING  |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING  |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING  |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5090) [1, 14000] loss: 0.314
(pid=5086) [1, 20000] loss: 0.196
(pid=5088) [1, 20000] loss: 0.155
(pid=5090) [1, 16000] loss: 0.279
(pid=5085) [5,  2000] loss: 1.501
Result for DEFAULT_f201f_00000:
  accuracy: 0.4442
  date: 2021-07-29_14-15-34
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 5
  loss: 1.5668423042297364
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 209.80427646636963
  time_this_iter_s: 40.83666944503784
  time_total_s: 209.80427646636963
  timestamp: 1627568134
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING  | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.56684 |     0.4442 |                    5 |
| DEFAULT_f201f_00001 | RUNNING  |                 |            2 |  128 |    8 | 0.000447305 |         |            |                      |
| DEFAULT_f201f_00002 | RUNNING  |                 |            2 |  256 |  256 | 0.0119053   |         |            |                      |
| DEFAULT_f201f_00003 | RUNNING  |                 |            2 |   32 |   16 | 0.00404483  |         |            |                      |
| DEFAULT_f201f_00004 | PENDING  |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING  |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING  |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING  |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING  |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING  |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5090) [1, 18000] loss: 0.237
Result for DEFAULT_f201f_00003:
  accuracy: 0.2865
  date: 2021-07-29_14-15-43
  done: true
  experiment_id: 8ff027eec3f14b2b9dc5ce8d11408ff8
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 1.899737369287014
  node_ip: 172.17.0.4
  pid: 5086
  should_checkpoint: true
  time_since_restore: 218.76438760757446
  time_this_iter_s: 218.76438760757446
  time_total_s: 218.76438760757446
  timestamp: 1627568143
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00003

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.7504788121402264
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING  | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.56684 |     0.4442 |                    5 |
| DEFAULT_f201f_00001 | RUNNING  |                 |            2 |  128 |    8 | 0.000447305 |         |            |                      |
| DEFAULT_f201f_00002 | RUNNING  |                 |            2 |  256 |  256 | 0.0119053   |         |            |                      |
| DEFAULT_f201f_00003 | RUNNING  | 172.17.0.4:5086 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | PENDING  |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING  |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING  |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING  |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING  |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING  |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5083) Files already downloaded and verified
(pid=5083) Files already downloaded and verified
Result for DEFAULT_f201f_00001:
  accuracy: 0.4357
  date: 2021-07-29_14-15-48
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 1.541055219617486
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 224.23737812042236
  time_this_iter_s: 224.23737812042236
  time_total_s: 224.23737812042236
  timestamp: 1627568148
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING    | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.56684 |     0.4442 |                    5 |
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.54106 |     0.4357 |                    1 |
| DEFAULT_f201f_00002 | RUNNING    |                 |            2 |  256 |  256 | 0.0119053   |         |            |                      |
| DEFAULT_f201f_00004 | RUNNING    |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5090) [1, 20000] loss: 0.212
(pid=5085) [6,  2000] loss: 1.502
(pid=5083) [1,  2000] loss: 2.252
(pid=5088) [2,  2000] loss: 1.505
Result for DEFAULT_f201f_00000:
  accuracy: 0.4722
  date: 2021-07-29_14-16-14
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 6
  loss: 1.5356089867591858
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 250.69555974006653
  time_this_iter_s: 40.8912832736969
  time_total_s: 250.69555974006653
  timestamp: 1627568174
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6012202549934387
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING    | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.53561 |     0.4722 |                    6 |
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.54106 |     0.4357 |                    1 |
| DEFAULT_f201f_00002 | RUNNING    |                 |            2 |  256 |  256 | 0.0119053   |         |            |                      |
| DEFAULT_f201f_00004 | RUNNING    |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5083) [1,  4000] loss: 0.963
(pid=5088) [2,  4000] loss: 0.749
Result for DEFAULT_f201f_00002:
  accuracy: 0.1789
  date: 2021-07-29_14-16-32
  done: true
  experiment_id: 9a45bdb8086e4379905275a77cbba199
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 2.121293088769913
  node_ip: 172.17.0.4
  pid: 5090
  should_checkpoint: true
  time_since_restore: 268.7365050315857
  time_this_iter_s: 268.7365050315857
  time_total_s: 268.7365050315857
  timestamp: 1627568192
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00002

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.7504788121402264
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING    | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.53561 |     0.4722 |                    6 |
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.54106 |     0.4357 |                    1 |
| DEFAULT_f201f_00002 | RUNNING    | 172.17.0.4:5090 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00004 | RUNNING    |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | PENDING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5087) Files already downloaded and verified
(pid=5087) Files already downloaded and verified
(pid=5083) [1,  6000] loss: 0.580
(pid=5088) [2,  6000] loss: 0.491
(pid=5085) [7,  2000] loss: 1.507
(pid=5083) [1,  8000] loss: 0.414
(pid=5087) [1,  2000] loss: 2.196
Result for DEFAULT_f201f_00000:
  accuracy: 0.4847
  date: 2021-07-29_14-17-00
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 7
  loss: 1.5526806345939637
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 296.0717935562134
  time_this_iter_s: 45.37623381614685
  time_total_s: 296.0717935562134
  timestamp: 1627568220
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.7504788121402264
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING    | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.55268 |     0.4847 |                    7 |
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.54106 |     0.4357 |                    1 |
| DEFAULT_f201f_00004 | RUNNING    |                 |            4 |    8 |   16 | 0.00134526  |         |            |                      |
| DEFAULT_f201f_00005 | RUNNING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [2,  8000] loss: 0.365
(pid=5083) [1, 10000] loss: 0.318
(pid=5087) [1,  4000] loss: 0.984
(pid=5088) [2, 10000] loss: 0.292
(pid=5085) [8,  2000] loss: 1.514
Result for DEFAULT_f201f_00004:
  accuracy: 0.3837
  date: 2021-07-29_14-17-31
  done: false
  experiment_id: 29b8f76bd84947dda7dcc9751ba1012d
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 1.6751174277424812
  node_ip: 172.17.0.4
  pid: 5083
  should_checkpoint: true
  time_since_restore: 106.32377362251282
  time_this_iter_s: 106.32377362251282
  time_total_s: 106.32377362251282
  timestamp: 1627568251
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00004

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING    | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.55268 |     0.4847 |                    7 |
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.54106 |     0.4357 |                    1 |
| DEFAULT_f201f_00004 | RUNNING    | 172.17.0.4:5083 |            4 |    8 |   16 | 0.00134526  | 1.67512 |     0.3837 |                    1 |
| DEFAULT_f201f_00005 | RUNNING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5087) [1,  6000] loss: 0.603
(pid=5088) [2, 12000] loss: 0.236
Result for DEFAULT_f201f_00000:
  accuracy: 0.4122
  date: 2021-07-29_14-17-41
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 8
  loss: 1.6352525778770446
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 336.84724020957947
  time_this_iter_s: 40.77544665336609
  time_total_s: 336.84724020957947
  timestamp: 1627568261
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING    | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.63525 |     0.4122 |                    8 |
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.54106 |     0.4357 |                    1 |
| DEFAULT_f201f_00004 | RUNNING    | 172.17.0.4:5083 |            4 |    8 |   16 | 0.00134526  | 1.67512 |     0.3837 |                    1 |
| DEFAULT_f201f_00005 | RUNNING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5087) [1,  8000] loss: 0.435
(pid=5083) [2,  2000] loss: 1.556
(pid=5088) [2, 14000] loss: 0.202
(pid=5087) [1, 10000] loss: 0.343
(pid=5085) [9,  2000] loss: 1.508
(pid=5088) [2, 16000] loss: 0.173
(pid=5083) [2,  4000] loss: 0.756
Result for DEFAULT_f201f_00000:
  accuracy: 0.4686
  date: 2021-07-29_14-18-23
  done: false
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 9
  loss: 1.5646762167930603
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 378.9824676513672
  time_this_iter_s: 42.13522744178772
  time_total_s: 378.9824676513672
  timestamp: 1627568303
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 16.0/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING    | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.56468 |     0.4686 |                    9 |
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.54106 |     0.4357 |                    1 |
| DEFAULT_f201f_00004 | RUNNING    | 172.17.0.4:5083 |            4 |    8 |   16 | 0.00134526  | 1.67512 |     0.3837 |                    1 |
| DEFAULT_f201f_00005 | RUNNING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5087) [1, 12000] loss: 0.276
(pid=5083) [2,  6000] loss: 0.508
(pid=5088) [2, 18000] loss: 0.153
(pid=5087) [1, 14000] loss: 0.231
(pid=5083) [2,  8000] loss: 0.368
(pid=5085) [10,  2000] loss: 1.511
(pid=5088) [2, 20000] loss: 0.138
Result for DEFAULT_f201f_00000:
  accuracy: 0.4533
  date: 2021-07-29_14-19-04
  done: true
  experiment_id: 519e73cc629f4a16ad544ba419c613fd
  hostname: 9ec45cf3ba83
  iterations_since_restore: 10
  loss: 1.6734770732879638
  node_ip: 172.17.0.4
  pid: 5085
  should_checkpoint: true
  time_since_restore: 420.25495433807373
  time_this_iter_s: 41.27248668670654
  time_total_s: 420.25495433807373
  timestamp: 1627568344
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: f201f_00000

== Status ==
Memory usage on this node: 16.0/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6653225509643554 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | RUNNING    | 172.17.0.4:5085 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.54106 |     0.4357 |                    1 |
| DEFAULT_f201f_00004 | RUNNING    | 172.17.0.4:5083 |            4 |    8 |   16 | 0.00134526  | 1.67512 |     0.3837 |                    1 |
| DEFAULT_f201f_00005 | RUNNING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | PENDING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5087) [1, 16000] loss: 0.201
(pid=5089) Files already downloaded and verified
(pid=5089) Files already downloaded and verified
(pid=5083) [2, 10000] loss: 0.296
(pid=5087) [1, 18000] loss: 0.177
Result for DEFAULT_f201f_00001:
  accuracy: 0.4878
  date: 2021-07-29_14-19-25
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 2
  loss: 1.4001507631493733
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 441.5508849620819
  time_this_iter_s: 217.31350684165955
  time_total_s: 441.5508849620819
  timestamp: 1627568365
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.5327366570568643 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.40015 |     0.4878 |                    2 |
| DEFAULT_f201f_00004 | RUNNING    | 172.17.0.4:5083 |            4 |    8 |   16 | 0.00134526  | 1.67512 |     0.3837 |                    1 |
| DEFAULT_f201f_00005 | RUNNING    |                 |            2 |   16 |   16 | 0.00126487  |         |            |                      |
| DEFAULT_f201f_00006 | RUNNING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | PENDING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_f201f_00004:
  accuracy: 0.4063
  date: 2021-07-29_14-19-28
  done: true
  experiment_id: 29b8f76bd84947dda7dcc9751ba1012d
  hostname: 9ec45cf3ba83
  iterations_since_restore: 2
  loss: 1.6531679716050625
  node_ip: 172.17.0.4
  pid: 5083
  should_checkpoint: true
  time_since_restore: 223.1200053691864
  time_this_iter_s: 116.79623174667358
  time_total_s: 223.1200053691864
  timestamp: 1627568368
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: f201f_00004

(pid=5084) Files already downloaded and verified
(pid=5089) [1,  2000] loss: 2.342
(pid=5084) Files already downloaded and verified
(pid=5087) [1, 20000] loss: 0.157
(pid=5088) [3,  2000] loss: 1.328
(pid=5089) [1,  4000] loss: 1.175
(pid=5084) [1,  2000] loss: 2.300
(pid=5088) [3,  4000] loss: 0.666
(pid=5089) [1,  6000] loss: 0.782
(pid=5084) [1,  4000] loss: 1.112
Result for DEFAULT_f201f_00005:
  accuracy: 0.4437
  date: 2021-07-29_14-20-13
  done: false
  experiment_id: 0aba26e190ca42cba089c32a9006ff74
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 1.5085618595972656
  node_ip: 172.17.0.4
  pid: 5087
  should_checkpoint: true
  time_since_restore: 218.37232851982117
  time_this_iter_s: 218.37232851982117
  time_total_s: 218.37232851982117
  timestamp: 1627568413
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00005

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6531679716050625 | Iter 1.000: -1.63816884136796
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.40015 |     0.4878 |                    2 |
| DEFAULT_f201f_00005 | RUNNING    | 172.17.0.4:5087 |            2 |   16 |   16 | 0.00126487  | 1.50856 |     0.4437 |                    1 |
| DEFAULT_f201f_00006 | RUNNING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | RUNNING    |                 |            8 |   64 |  256 | 0.000285913 |         |            |                      |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [3,  6000] loss: 0.432
(pid=5089) [1,  8000] loss: 0.587
(pid=5087) [2,  2000] loss: 1.513
Result for DEFAULT_f201f_00007:
  accuracy: 0.2903
  date: 2021-07-29_14-20-35
  done: true
  experiment_id: b17549cee3d041f19ecfddc572127732
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 1.9859605166435241
  node_ip: 172.17.0.4
  pid: 5084
  should_checkpoint: true
  time_since_restore: 65.69195485115051
  time_this_iter_s: 65.69195485115051
  time_total_s: 65.69195485115051
  timestamp: 1627568435
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00007

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6531679716050625 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.40015 |     0.4878 |                    2 |
| DEFAULT_f201f_00005 | RUNNING    | 172.17.0.4:5087 |            2 |   16 |   16 | 0.00126487  | 1.50856 |     0.4437 |                    1 |
| DEFAULT_f201f_00006 | RUNNING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00007 | RUNNING    | 172.17.0.4:5084 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00008 | PENDING    |                 |            8 |   32 |   16 | 0.00172471  |         |            |                      |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [3,  8000] loss: 0.324
(pid=8726) Files already downloaded and verified
(pid=8726) Files already downloaded and verified
(pid=5089) [1, 10000] loss: 0.469
(pid=5087) [2,  4000] loss: 0.768
(pid=5088) [3, 10000] loss: 0.257
(pid=5089) [1, 12000] loss: 0.392
(pid=8726) [1,  2000] loss: 2.275
(pid=5087) [2,  6000] loss: 0.506
(pid=5088) [3, 12000] loss: 0.218
(pid=5087) [2,  8000] loss: 0.377
(pid=8726) [1,  4000] loss: 0.945
(pid=5089) [1, 14000] loss: 0.336
(pid=5088) [3, 14000] loss: 0.182
(pid=5087) [2, 10000] loss: 0.303
(pid=5088) [3, 16000] loss: 0.158
Result for DEFAULT_f201f_00008:
  accuracy: 0.4047
  date: 2021-07-29_14-21-45
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 1.620893050098419
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 66.44279766082764
  time_this_iter_s: 66.44279766082764
  time_total_s: 66.44279766082764
  timestamp: 1627568505
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.6531679716050625 | Iter 1.000: -1.64800523892045
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.40015 |     0.4878 |                    2 |
| DEFAULT_f201f_00005 | RUNNING    | 172.17.0.4:5087 |            2 |   16 |   16 | 0.00126487  | 1.50856 |     0.4437 |                    1 |
| DEFAULT_f201f_00006 | RUNNING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.62089 |     0.4047 |                    1 |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5089) [1, 16000] loss: 0.293
(pid=5087) [2, 12000] loss: 0.250
(pid=5088) [3, 18000] loss: 0.142
(pid=5087) [2, 14000] loss: 0.210
(pid=8726) [2,  2000] loss: 1.533
(pid=5089) [1, 18000] loss: 0.261
(pid=5088) [3, 20000] loss: 0.130
(pid=5087) [2, 16000] loss: 0.189
(pid=5089) [1, 20000] loss: 0.235
(pid=8726) [2,  4000] loss: 0.731
(pid=5087) [2, 18000] loss: 0.167
Result for DEFAULT_f201f_00008:
  accuracy: 0.4996
  date: 2021-07-29_14-22-48
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 2
  loss: 1.3640813674926757
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 129.95181798934937
  time_this_iter_s: 63.50902032852173
  time_total_s: 129.95181798934937
  timestamp: 1627568568
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 16.0/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.526659367377218 | Iter 1.000: -1.64800523892045
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.40015 |     0.4878 |                    2 |
| DEFAULT_f201f_00005 | RUNNING    | 172.17.0.4:5087 |            2 |   16 |   16 | 0.00126487  | 1.50856 |     0.4437 |                    1 |
| DEFAULT_f201f_00006 | RUNNING    |                 |            2 |  256 |   16 | 0.0404286   |         |            |                      |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.36408 |     0.4996 |                    2 |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5087) [2, 20000] loss: 0.151
Result for DEFAULT_f201f_00001:
  accuracy: 0.5397
  date: 2021-07-29_14-22-53
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 3
  loss: 1.2915692168400623
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 649.3570704460144
  time_this_iter_s: 207.8061854839325
  time_total_s: 649.3570704460144
  timestamp: 1627568573
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: f201f_00001

Result for DEFAULT_f201f_00006:
  accuracy: 0.0956
  date: 2021-07-29_14-23-00
  done: true
  experiment_id: 7dc069aa9c2144ee9e8e492ed6b3f78a
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 2.3285563125133515
  node_ip: 172.17.0.4
  pid: 5089
  should_checkpoint: true
  time_since_restore: 233.9469187259674
  time_this_iter_s: 233.9469187259674
  time_total_s: 233.9469187259674
  timestamp: 1627568580
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00006

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.526659367377218 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.29157 |     0.5397 |                    3 |
| DEFAULT_f201f_00005 | RUNNING    | 172.17.0.4:5087 |            2 |   16 |   16 | 0.00126487  | 1.50856 |     0.4437 |                    1 |
| DEFAULT_f201f_00006 | RUNNING    | 172.17.0.4:5089 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.36408 |     0.4996 |                    2 |
| DEFAULT_f201f_00009 | PENDING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=9546) Files already downloaded and verified
(pid=9546) Files already downloaded and verified
(pid=8726) [3,  2000] loss: 1.361
(pid=5088) [4,  2000] loss: 1.217
Result for DEFAULT_f201f_00005:
  accuracy: 0.4488
  date: 2021-07-29_14-23-27
  done: true
  experiment_id: 0aba26e190ca42cba089c32a9006ff74
  hostname: 9ec45cf3ba83
  iterations_since_restore: 2
  loss: 1.5376347200442106
  node_ip: 172.17.0.4
  pid: 5087
  should_checkpoint: true
  time_since_restore: 411.8736696243286
  time_this_iter_s: 193.50134110450745
  time_total_s: 411.8736696243286
  timestamp: 1627568607
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: f201f_00005

== Status ==
Memory usage on this node: 15.9/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.6751174277424812
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.29157 |     0.5397 |                    3 |
| DEFAULT_f201f_00005 | RUNNING    | 172.17.0.4:5087 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.36408 |     0.4996 |                    2 |
| DEFAULT_f201f_00009 | RUNNING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [4,  4000] loss: 0.596
(pid=9546) [1,  2000] loss: 2.319
(pid=8726) [3,  4000] loss: 0.668
(pid=5088) [4,  6000] loss: 0.405
(pid=9546) [1,  4000] loss: 1.160
Result for DEFAULT_f201f_00008:
  accuracy: 0.529
  date: 2021-07-29_14-23-50
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 3
  loss: 1.320628886294365
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 192.0822982788086
  time_this_iter_s: 62.13048028945923
  time_total_s: 192.0822982788086
  timestamp: 1627568630
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 15.4/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.6751174277424812
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.29157 |     0.5397 |                    3 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.32063 |     0.529  |                    3 |
| DEFAULT_f201f_00009 | RUNNING    |                 |            8 |   64 |   16 | 0.0574224   |         |            |                      |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [4,  8000] loss: 0.297
Result for DEFAULT_f201f_00009:
  accuracy: 0.0983
  date: 2021-07-29_14-24-06
  done: true
  experiment_id: e3e77385e2d9496db44502e89f7563e3
  hostname: 9ec45cf3ba83
  iterations_since_restore: 1
  loss: 2.319805237197876
  node_ip: 172.17.0.4
  pid: 9546
  should_checkpoint: true
  time_since_restore: 64.11058259010315
  time_this_iter_s: 64.11058259010315
  time_total_s: 64.11058259010315
  timestamp: 1627568646
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f201f_00009

== Status ==
Memory usage on this node: 15.4/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.6066401032447815 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.29157 |     0.5397 |                    3 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.32063 |     0.529  |                    3 |
| DEFAULT_f201f_00009 | RUNNING    | 172.17.0.4:9546 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=8726) [4,  2000] loss: 1.258
(pid=5088) [4, 10000] loss: 0.241
(pid=8726) [4,  4000] loss: 0.631
(pid=5088) [4, 12000] loss: 0.197
Result for DEFAULT_f201f_00008:
  accuracy: 0.5519
  date: 2021-07-29_14-24-40
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 4
  loss: 1.273954435968399
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 242.25610089302063
  time_this_iter_s: 50.173802614212036
  time_total_s: 242.25610089302063
  timestamp: 1627568680
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.44029726960659 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.29157 |     0.5397 |                    3 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.27395 |     0.5519 |                    4 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [4, 14000] loss: 0.170
(pid=8726) [5,  2000] loss: 1.193
(pid=5088) [4, 16000] loss: 0.152
(pid=8726) [5,  4000] loss: 0.607
(pid=5088) [4, 18000] loss: 0.134
Result for DEFAULT_f201f_00008:
  accuracy: 0.5585
  date: 2021-07-29_14-25-34
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 5
  loss: 1.2658138296961785
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 296.21075320243835
  time_this_iter_s: 53.954652309417725
  time_total_s: 296.21075320243835
  timestamp: 1627568734
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.44029726960659 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.29157 |     0.5397 |                    3 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.26581 |     0.5585 |                    5 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [4, 20000] loss: 0.117
(pid=8726) [6,  2000] loss: 1.159
(pid=8726) [6,  4000] loss: 0.583
Result for DEFAULT_f201f_00001:
  accuracy: 0.5831
  date: 2021-07-29_14-26-17
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 4
  loss: 1.1855550648670643
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 853.2846131324768
  time_this_iter_s: 203.9275426864624
  time_total_s: 853.2846131324768
  timestamp: 1627568777
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.18556 |     0.5831 |                    4 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.26581 |     0.5585 |                    5 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_f201f_00008:
  accuracy: 0.5757
  date: 2021-07-29_14-26-31
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 6
  loss: 1.213883140027523
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 352.9563558101654
  time_this_iter_s: 56.74560260772705
  time_total_s: 352.9563558101654
  timestamp: 1627568791
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.18556 |     0.5831 |                    4 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.21388 |     0.5757 |                    6 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [5,  2000] loss: 1.107
(pid=5088) [5,  4000] loss: 0.575
(pid=8726) [7,  2000] loss: 1.115
(pid=5088) [5,  6000] loss: 0.372
(pid=8726) [7,  4000] loss: 0.568
(pid=5088) [5,  8000] loss: 0.280
Result for DEFAULT_f201f_00008:
  accuracy: 0.5837
  date: 2021-07-29_14-27-20
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 7
  loss: 1.1828787779331207
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 401.7406098842621
  time_this_iter_s: 48.78425407409668
  time_total_s: 401.7406098842621
  timestamp: 1627568840
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.6352525778770446 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.18556 |     0.5831 |                    4 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.18288 |     0.5837 |                    7 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [5, 10000] loss: 0.230
(pid=8726) [8,  2000] loss: 1.091
(pid=5088) [5, 12000] loss: 0.187
(pid=5088) [5, 14000] loss: 0.159
(pid=8726) [8,  4000] loss: 0.553
(pid=5088) [5, 16000] loss: 0.141
Result for DEFAULT_f201f_00008:
  accuracy: 0.5696
  date: 2021-07-29_14-28-16
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 8
  loss: 1.2416244268774985
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 458.10355401039124
  time_this_iter_s: 56.36294412612915
  time_total_s: 458.10355401039124
  timestamp: 1627568896
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.18556 |     0.5831 |                    4 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.24162 |     0.5696 |                    8 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [5, 18000] loss: 0.127
(pid=8726) [9,  2000] loss: 1.074
(pid=5088) [5, 20000] loss: 0.113
(pid=8726) [9,  4000] loss: 0.545
Result for DEFAULT_f201f_00008:
  accuracy: 0.5852
  date: 2021-07-29_14-29-06
  done: false
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 9
  loss: 1.1980089530587197
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 508.1109640598297
  time_this_iter_s: 50.00741004943848
  time_total_s: 508.1109640598297
  timestamp: 1627568946
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.18556 |     0.5831 |                    4 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.19801 |     0.5852 |                    9 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_f201f_00001:
  accuracy: 0.5986
  date: 2021-07-29_14-29-13
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 5
  loss: 1.15476273704404
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 1029.5660145282745
  time_this_iter_s: 176.28140139579773
  time_total_s: 1029.5660145282745
  timestamp: 1627568953
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.15476 |     0.5986 |                    5 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.19801 |     0.5852 |                    9 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=8726) [10,  2000] loss: 1.034
(pid=5088) [6,  2000] loss: 1.071
(pid=8726) [10,  4000] loss: 0.538
(pid=5088) [6,  4000] loss: 0.515
Result for DEFAULT_f201f_00008:
  accuracy: 0.5787
  date: 2021-07-29_14-29-53
  done: true
  experiment_id: 828c343465b746a6833f7a13f52e8c21
  hostname: 9ec45cf3ba83
  iterations_since_restore: 10
  loss: 1.2242828347325325
  node_ip: 172.17.0.4
  pid: 8726
  should_checkpoint: true
  time_since_restore: 554.6373965740204
  time_this_iter_s: 46.526432514190674
  time_total_s: 554.6373965740204
  timestamp: 1627568993
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: f201f_00008

== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.15476 |     0.5986 |                    5 |
| DEFAULT_f201f_00008 | RUNNING    | 172.17.0.4:8726 |            8 |   32 |   16 | 0.00172471  | 1.22428 |     0.5787 |                   10 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [6,  6000] loss: 0.354
(pid=5088) [6,  8000] loss: 0.259
(pid=5088) [6, 10000] loss: 0.218
(pid=5088) [6, 12000] loss: 0.179
(pid=5088) [6, 14000] loss: 0.156
(pid=5088) [6, 16000] loss: 0.136
(pid=5088) [6, 18000] loss: 0.116
(pid=5088) [6, 20000] loss: 0.108
Result for DEFAULT_f201f_00001:
  accuracy: 0.5862
  date: 2021-07-29_14-31-07
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 6
  loss: 1.1807675102452049
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 1143.6345767974854
  time_this_iter_s: 114.06856226921082
  time_total_s: 1143.6345767974854
  timestamp: 1627569067
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.18077 |     0.5862 |                    6 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00008 | TERMINATED |                 |            8 |   32 |   16 | 0.00172471  | 1.22428 |     0.5787 |                   10 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [7,  2000] loss: 0.969
(pid=5088) [7,  4000] loss: 0.489
(pid=5088) [7,  6000] loss: 0.332
(pid=5088) [7,  8000] loss: 0.258
(pid=5088) [7, 10000] loss: 0.207
(pid=5088) [7, 12000] loss: 0.170
(pid=5088) [7, 14000] loss: 0.144
(pid=5088) [7, 16000] loss: 0.130
(pid=5088) [7, 18000] loss: 0.114
(pid=5088) [7, 20000] loss: 0.104
Result for DEFAULT_f201f_00001:
  accuracy: 0.5818
  date: 2021-07-29_14-32-42
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 7
  loss: 1.220972312376625
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 1238.2733232975006
  time_this_iter_s: 94.63874650001526
  time_total_s: 1238.2733232975006
  timestamp: 1627569162
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.4384385023772714 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.22097 |     0.5818 |                    7 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00008 | TERMINATED |                 |            8 |   32 |   16 | 0.00172471  | 1.22428 |     0.5787 |                   10 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [8,  2000] loss: 0.954
(pid=5088) [8,  4000] loss: 0.465
(pid=5088) [8,  6000] loss: 0.314
(pid=5088) [8,  8000] loss: 0.246
(pid=5088) [8, 10000] loss: 0.196
(pid=5088) [8, 12000] loss: 0.161
(pid=5088) [8, 14000] loss: 0.142
(pid=5088) [8, 16000] loss: 0.124
(pid=5088) [8, 18000] loss: 0.108
(pid=5088) [8, 20000] loss: 0.098
Result for DEFAULT_f201f_00001:
  accuracy: 0.5836
  date: 2021-07-29_14-34-13
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 8
  loss: 1.246836426538159
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 1329.7241804599762
  time_this_iter_s: 91.45085716247559
  time_total_s: 1329.7241804599762
  timestamp: 1627569253
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.246836426538159 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.24684 |     0.5836 |                    8 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00008 | TERMINATED |                 |            8 |   32 |   16 | 0.00172471  | 1.22428 |     0.5787 |                   10 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [9,  2000] loss: 0.924
(pid=5088) [9,  4000] loss: 0.440
(pid=5088) [9,  6000] loss: 0.298
(pid=5088) [9,  8000] loss: 0.228
(pid=5088) [9, 10000] loss: 0.183
(pid=5088) [9, 12000] loss: 0.158
(pid=5088) [9, 14000] loss: 0.138
(pid=5088) [9, 16000] loss: 0.117
(pid=5088) [9, 18000] loss: 0.110
(pid=5088) [9, 20000] loss: 0.096
Result for DEFAULT_f201f_00001:
  accuracy: 0.5968
  date: 2021-07-29_14-35-49
  done: false
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 9
  loss: 1.2085561733988057
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 1425.0174763202667
  time_this_iter_s: 95.29329586029053
  time_total_s: 1425.0174763202667
  timestamp: 1627569349
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 15.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.246836426538159 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.20856 |     0.5968 |                    9 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00008 | TERMINATED |                 |            8 |   32 |   16 | 0.00172471  | 1.22428 |     0.5787 |                   10 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(pid=5088) [10,  2000] loss: 0.828
(pid=5088) [10,  4000] loss: 0.427
(pid=5088) [10,  6000] loss: 0.290
(pid=5088) [10,  8000] loss: 0.225
(pid=5088) [10, 10000] loss: 0.176
(pid=5088) [10, 12000] loss: 0.152
(pid=5088) [10, 14000] loss: 0.131
(pid=5088) [10, 16000] loss: 0.115
(pid=5088) [10, 18000] loss: 0.102
(pid=5088) [10, 20000] loss: 0.091
Result for DEFAULT_f201f_00001:
  accuracy: 0.5926
  date: 2021-07-29_14-37-26
  done: true
  experiment_id: 3f191cc7a1ef496990f3537841cc639f
  hostname: 9ec45cf3ba83
  iterations_since_restore: 10
  loss: 1.23822381478647
  node_ip: 172.17.0.4
  pid: 5088
  should_checkpoint: true
  time_since_restore: 1522.388427734375
  time_this_iter_s: 97.37095141410828
  time_total_s: 1522.388427734375
  timestamp: 1627569446
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: f201f_00001

== Status ==
Memory usage on this node: 15.1/31.3 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.246836426538159 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00001 | RUNNING    | 172.17.0.4:5088 |            2 |  128 |    8 | 0.000447305 | 1.23822 |     0.5926 |                   10 |
| DEFAULT_f201f_00000 | TERMINATED |                 |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |                 |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |                 |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |                 |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |                 |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |                 |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |                 |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00008 | TERMINATED |                 |            8 |   32 |   16 | 0.00172471  | 1.22428 |     0.5787 |                   10 |
| DEFAULT_f201f_00009 | TERMINATED |                 |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


== Status ==
Memory usage on this node: 14.9/31.3 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.246836426538159 | Iter 4.000: -1.273954435968399 | Iter 2.000: -1.5376347200442106 | Iter 1.000: -1.7874273985147475
Resources requested: 0/8 CPUs, 0/2 GPUs, 0.0/11.74 GiB heap, 0.0/5.87 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2021-07-29_14-12-02
Number of trials: 10/10 (10 TERMINATED)
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc   |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_f201f_00000 | TERMINATED |       |           16 |  128 |   16 | 0.0173062   | 1.67348 |     0.4533 |                   10 |
| DEFAULT_f201f_00001 | TERMINATED |       |            2 |  128 |    8 | 0.000447305 | 1.23822 |     0.5926 |                   10 |
| DEFAULT_f201f_00002 | TERMINATED |       |            2 |  256 |  256 | 0.0119053   | 2.12129 |     0.1789 |                    1 |
| DEFAULT_f201f_00003 | TERMINATED |       |            2 |   32 |   16 | 0.00404483  | 1.89974 |     0.2865 |                    1 |
| DEFAULT_f201f_00004 | TERMINATED |       |            4 |    8 |   16 | 0.00134526  | 1.65317 |     0.4063 |                    2 |
| DEFAULT_f201f_00005 | TERMINATED |       |            2 |   16 |   16 | 0.00126487  | 1.53763 |     0.4488 |                    2 |
| DEFAULT_f201f_00006 | TERMINATED |       |            2 |  256 |   16 | 0.0404286   | 2.32856 |     0.0956 |                    1 |
| DEFAULT_f201f_00007 | TERMINATED |       |            8 |   64 |  256 | 0.000285913 | 1.98596 |     0.2903 |                    1 |
| DEFAULT_f201f_00008 | TERMINATED |       |            8 |   32 |   16 | 0.00172471  | 1.22428 |     0.5787 |                   10 |
| DEFAULT_f201f_00009 | TERMINATED |       |            8 |   64 |   16 | 0.0574224   | 2.31981 |     0.0983 |                    1 |
+---------------------+------------+-------+--------------+------+------+-------------+---------+------------+----------------------+


Best trial config: {'l1': 32, 'l2': 16, 'lr': 0.001724709016268422, 'batch_size': 8}
Best trial final validation loss: 1.2242828347325325
Best trial final validation accuracy: 0.5787
Files already downloaded and verified
Files already downloaded and verified
Best trial test set accuracy: 0.5797

If you run the code, an example output could look like this:

Number of trials: 10 (10 TERMINATED)
+-----+------+------+-------------+--------------+---------+------------+--------------------+
| ... |   l1 |   l2 |          lr |   batch_size |    loss |   accuracy | training_iteration |
|-----+------+------+-------------+--------------+---------+------------+--------------------|
| ... |   64 |    4 | 0.00011629  |            2 | 1.87273 |     0.244  |                  2 |
| ... |   32 |   64 | 0.000339763 |            8 | 1.23603 |     0.567  |                  8 |
| ... |    8 |   16 | 0.00276249  |           16 | 1.1815  |     0.5836 |                 10 |
| ... |    4 |   64 | 0.000648721 |            4 | 1.31131 |     0.5224 |                  8 |
| ... |   32 |   16 | 0.000340753 |            8 | 1.26454 |     0.5444 |                  8 |
| ... |    8 |    4 | 0.000699775 |            8 | 1.99594 |     0.1983 |                  2 |
| ... |  256 |    8 | 0.0839654   |           16 | 2.3119  |     0.0993 |                  1 |
| ... |   16 |  128 | 0.0758154   |           16 | 2.33575 |     0.1327 |                  1 |
| ... |   16 |    8 | 0.0763312   |           16 | 2.31129 |     0.1042 |                  4 |
| ... |  128 |   16 | 0.000124903 |            4 | 2.26917 |     0.1945 |                  1 |
+-----+------+------+-------------+--------------+---------+------------+--------------------+


Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
Best trial final validation loss: 1.181501
Best trial final validation accuracy: 0.5836
Best trial test set accuracy: 0.5806

Most trials have been stopped early in order to avoid wasting resources. The best performing trial achieved a validation accuracy of about 58%, which could be confirmed on the test set.

So that’s it! You can now tune the parameters of your PyTorch models.

Total running time of the script: ( 26 minutes 23.448 seconds)

Gallery generated by Sphinx-Gallery

공식 문서 (영어)

PyTorch 공식 문서입니다.

공식 문서로 이동

한국어 튜토리얼

한국어로 번역 중인 PyTorch 튜토리얼입니다.

튜토리얼로 이동

커뮤니티

다른 사용자들과 의견을 나눠보세요!

커뮤니티로 이동