Hyperparameter tuning with Ray Tune

Hyperparameter tuning can make the difference between an average model and a highly accurate one. Often simple things like choosing a different learning rate or changing a network layer size can have a dramatic impact on your model performance.

Fortunately, there are tools that help with finding the best combination of parameters. Ray Tune is an industry standard tool for distributed hyperparameter tuning. Ray Tune includes the latest hyperparameter search algorithms, integrates with TensorBoard and other analysis libraries, and natively supports distributed training through Ray’s distributed machine learning engine.

In this tutorial, we will show you how to integrate Ray Tune into your PyTorch training workflow. We will extend this tutorial from the PyTorch documentation for training a CIFAR10 image classifier.

As you will see, we only need to add some slight modifications. In particular, we need to

1. wrap data loading and training in functions,

2. make some network parameters configurable,

3. add checkpointing (optional),

4. and define the search space for the model tuning

To run this tutorial, please make sure the following packages are installed:

• ray[tune]: Distributed hyperparameter tuning library

• torchvision: For the data transformers

Setup / Imports

Let’s start with the imports:

from functools import partial
import numpy as np
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import random_split
import torchvision
import torchvision.transforms as transforms
from ray import tune
from ray.tune import CLIReporter
from ray.tune.schedulers import ASHAScheduler

Most of the imports are needed for building the PyTorch model. Only the last three imports are for Ray Tune.

Data loaders

We wrap the data loaders in their own function and pass a global data directory. This way we can share a data directory between different trials.

def load_data(data_dir="./data"):
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

    trainset = torchvision.datasets.CIFAR10(
        root=data_dir, train=True, download=True, transform=transform)

    testset = torchvision.datasets.CIFAR10(
        root=data_dir, train=False, download=True, transform=transform)

    return trainset, testset

Configurable neural network

We can only tune those parameters that are configurable. In this example, we can specify the layer sizes of the fully connected layers:

class Net(nn.Module):
    def __init__(self, l1=120, l2=84):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, l1)
        self.fc2 = nn.Linear(l1, l2)
        self.fc3 = nn.Linear(l2, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The train function

Now it gets interesting, because we introduce some changes to the example from the PyTorch documentation.

We wrap the training script in a function train_cifar(config, checkpoint_dir=None, data_dir=None). As you can guess, the config parameter will receive the hyperparameters we would like to train with. The checkpoint_dir parameter is used to restore checkpoints. The data_dir specifies the directory where we load and store the data, so multiple runs can share the same data source.

net = Net(config["l1"], config["l2"])

if checkpoint_dir:
    model_state, optimizer_state = torch.load(
        os.path.join(checkpoint_dir, "checkpoint"))
    net.load_state_dict(model_state)
    optimizer.load_state_dict(optimizer_state)

The learning rate of the optimizer is made configurable, too:

optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

We also split the training data into a training and validation subset. We thus train on 80% of the data and calculate the validation loss on the remaining 20%. The batch sizes with which we iterate through the training and test sets are configurable as well.

Adding (multi) GPU support with DataParallel

Image classification benefits largely from GPUs. Luckily, we can continue to use PyTorch’s abstractions in Ray Tune. Thus, we can wrap our model in nn.DataParallel to support data parallel training on multiple GPUs:

device = "cpu"
if torch.cuda.is_available():
    device = "cuda:0"
    if torch.cuda.device_count() > 1:
        net = nn.DataParallel(net)
net.to(device)

By using a device variable we make sure that training also works when we have no GPUs available. PyTorch requires us to send our data to the GPU memory explicitly, like this:

for i, data in enumerate(trainloader, 0):
    inputs, labels = data
    inputs, labels = inputs.to(device), labels.to(device)

The code now supports training on CPUs, on a single GPU, and on multiple GPUs. Notably, Ray also supports fractional GPUs so we can share GPUs among trials, as long as the model still fits on the GPU memory. We’ll come back to that later.

Communicating with Ray Tune

The most interesting part is the communication with Ray Tune:

with tune.checkpoint_dir(epoch) as checkpoint_dir:
    path = os.path.join(checkpoint_dir, "checkpoint")
    torch.save((net.state_dict(), optimizer.state_dict()), path)

tune.report(loss=(val_loss / val_steps), accuracy=correct / total)

Here we first save a checkpoint and then report some metrics back to Ray Tune. Specifically, we send the validation loss and accuracy back to Ray Tune. Ray Tune can then use these metrics to decide which hyperparameter configuration lead to the best results. These metrics can also be used to stop bad performing trials early in order to avoid wasting resources on those trials.

The checkpoint saving is optional, however, it is necessary if we wanted to use advanced schedulers like Population Based Training. Also, by saving the checkpoint we can later load the trained models and validate them on a test set.

Full training function

The full code example looks like this:

def train_cifar(config, checkpoint_dir=None, data_dir=None):
    net = Net(config["l1"], config["l2"])

    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if torch.cuda.device_count() > 1:
            net = nn.DataParallel(net)
    net.to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

    if checkpoint_dir:
        model_state, optimizer_state = torch.load(
            os.path.join(checkpoint_dir, "checkpoint"))
        net.load_state_dict(model_state)
        optimizer.load_state_dict(optimizer_state)

    trainset, testset = load_data(data_dir)

    test_abs = int(len(trainset) * 0.8)
    train_subset, val_subset = random_split(
        trainset, [test_abs, len(trainset) - test_abs])

    trainloader = torch.utils.data.DataLoader(
        train_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)
    valloader = torch.utils.data.DataLoader(
        val_subset,
        batch_size=int(config["batch_size"]),
        shuffle=True,
        num_workers=8)

    for epoch in range(10):  # loop over the dataset multiple times
        running_loss = 0.0
        epoch_steps = 0
        for i, data in enumerate(trainloader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            epoch_steps += 1
            if i % 2000 == 1999:  # print every 2000 mini-batches
                print("[%d, %5d] loss: %.3f" % (epoch + 1, i + 1,
                                                running_loss / epoch_steps))
                running_loss = 0.0

        # Validation loss
        val_loss = 0.0
        val_steps = 0
        total = 0
        correct = 0
        for i, data in enumerate(valloader, 0):
            with torch.no_grad():
                inputs, labels = data
                inputs, labels = inputs.to(device), labels.to(device)

                outputs = net(inputs)
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()

                loss = criterion(outputs, labels)
                val_loss += loss.cpu().numpy()
                val_steps += 1

        with tune.checkpoint_dir(epoch) as checkpoint_dir:
            path = os.path.join(checkpoint_dir, "checkpoint")
            torch.save((net.state_dict(), optimizer.state_dict()), path)

        tune.report(loss=(val_loss / val_steps), accuracy=correct / total)
    print("Finished Training")

As you can see, most of the code is adapted directly from the original example.

Test set accuracy

Commonly the performance of a machine learning model is tested on a hold-out test set with data that has not been used for training the model. We also wrap this in a function:

def test_accuracy(net, device="cpu"):
    trainset, testset = load_data()

    testloader = torch.utils.data.DataLoader(
        testset, batch_size=4, shuffle=False, num_workers=2)

    correct = 0
    total = 0
    with torch.no_grad():
        for data in testloader:
            images, labels = data
            images, labels = images.to(device), labels.to(device)
            outputs = net(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    return correct / total

The function also expects a device parameter, so we can do the test set validation on a GPU.

Configuring the search space

Lastly, we need to define Ray Tune’s search space. Here is an example:

config = {
    "l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
    "lr": tune.loguniform(1e-4, 1e-1),
    "batch_size": tune.choice([2, 4, 8, 16])
}

The tune.sample_from() function makes it possible to define your own sample methods to obtain hyperparameters. In this example, the l1 and l2 parameters should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256. The lr (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly, the batch size is a choice between 2, 4, 8, and 16.

At each trial, Ray Tune will now randomly sample a combination of parameters from these search spaces. It will then train a number of models in parallel and find the best performing one among these. We also use the ASHAScheduler which will terminate bad performing trials early.

We wrap the train_cifar function with functools.partial to set the constant data_dir parameter. We can also tell Ray Tune what resources should be available for each trial:

gpus_per_trial = 2
# ...
result = tune.run(
    partial(train_cifar, data_dir=data_dir),
    resources_per_trial={"cpu": 8, "gpu": gpus_per_trial},
    config=config,
    num_samples=num_samples,
    scheduler=scheduler,
    progress_reporter=reporter,
    checkpoint_at_end=True)

You can specify the number of CPUs, which are then available e.g. to increase the num_workers of the PyTorch DataLoader instances. The selected number of GPUs are made visible to PyTorch in each trial. Trials do not have access to GPUs that haven’t been requested for them - so you don’t have to care about two trials using the same set of resources.

Here we can also specify fractional GPUs, so something like gpus_per_trial=0.5 is completely valid. The trials will then share GPUs among each other. You just have to make sure that the models still fit in the GPU memory.

After training the models, we will find the best performing one and load the trained network from the checkpoint file. We then obtain the test set accuracy and report everything by printing.

The full main function looks like this:

def main(num_samples=10, max_num_epochs=10, gpus_per_trial=2):
    data_dir = os.path.abspath("./data")
    load_data(data_dir)
    config = {
        "l1": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "l2": tune.sample_from(lambda _: 2 ** np.random.randint(2, 9)),
        "lr": tune.loguniform(1e-4, 1e-1),
        "batch_size": tune.choice([2, 4, 8, 16])
    }
    scheduler = ASHAScheduler(
        metric="loss",
        mode="min",
        max_t=max_num_epochs,
        grace_period=1,
        reduction_factor=2)
    reporter = CLIReporter(
        # parameter_columns=["l1", "l2", "lr", "batch_size"],
        metric_columns=["loss", "accuracy", "training_iteration"])
    result = tune.run(
        partial(train_cifar, data_dir=data_dir),
        resources_per_trial={"cpu": 2, "gpu": gpus_per_trial},
        config=config,
        num_samples=num_samples,
        scheduler=scheduler,
        progress_reporter=reporter)

    best_trial = result.get_best_trial("loss", "min", "last")
    print("Best trial config: {}".format(best_trial.config))
    print("Best trial final validation loss: {}".format(
        best_trial.last_result["loss"]))
    print("Best trial final validation accuracy: {}".format(
        best_trial.last_result["accuracy"]))

    best_trained_model = Net(best_trial.config["l1"], best_trial.config["l2"])
    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if gpus_per_trial > 1:
            best_trained_model = nn.DataParallel(best_trained_model)
    best_trained_model.to(device)

    best_checkpoint_dir = best_trial.checkpoint.value
    model_state, optimizer_state = torch.load(os.path.join(
        best_checkpoint_dir, "checkpoint"))
    best_trained_model.load_state_dict(model_state)

    test_acc = test_accuracy(best_trained_model, device)
    print("Best trial test set accuracy: {}".format(test_acc))


if __name__ == "__main__":
    # You can change the number of GPUs per trial here:
    main(num_samples=10, max_num_epochs=10, gpus_per_trial=0)

Out:

Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to /workspace/ko-latest/beginner_source/data/cifar-10-python.tar.gz
Extracting /workspace/ko-latest/beginner_source/data/cifar-10-python.tar.gz to /workspace/ko-latest/beginner_source/data
Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:45:29 (running for 00:00:00.20)
Memory usage on this node: 19.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (9 PENDING, 1 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 |
| DEFAULT_0c10e_00001 | PENDING  |                 |            4 |    4 |    4 | 0.073943    |
| DEFAULT_0c10e_00002 | PENDING  |                 |            8 |   32 |    8 | 0.000505412 |
| DEFAULT_0c10e_00003 | PENDING  |                 |            2 |   16 |   16 | 0.00365822  |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |
+---------------------+----------+-----------------+--------------+------+------+-------------+


(func pid=5006) Files already downloaded and verified
(func pid=5006) Files already downloaded and verified
(func pid=5009) Files already downloaded and verified
(func pid=5004) Files already downloaded and verified
(func pid=5008) Files already downloaded and verified
(func pid=5009) Files already downloaded and verified
(func pid=5004) Files already downloaded and verified
(func pid=5008) Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:45:34 (running for 00:00:05.25)
Memory usage on this node: 21.4/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |
+---------------------+----------+-----------------+--------------+------+------+-------------+


== Status ==
Current time: 2022-04-10 04:45:40 (running for 00:00:11.22)
Memory usage on this node: 21.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |
+---------------------+----------+-----------------+--------------+------+------+-------------+


(func pid=5004) [1,  2000] loss: 2.161
(func pid=5009) [1,  2000] loss: 2.344
(func pid=5006) [1,  2000] loss: 2.304
(func pid=5008) [1,  2000] loss: 2.280
== Status ==
Current time: 2022-04-10 04:45:45 (running for 00:00:16.24)
Memory usage on this node: 21.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |
+---------------------+----------+-----------------+--------------+------+------+-------------+


== Status ==
Current time: 2022-04-10 04:45:50 (running for 00:00:21.27)
Memory usage on this node: 21.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |
+---------------------+----------+-----------------+--------------+------+------+-------------+


(func pid=5004) [1,  4000] loss: 1.003
(func pid=5009) [1,  4000] loss: 1.171
(func pid=5006) [1,  4000] loss: 1.149
== Status ==
Current time: 2022-04-10 04:45:55 (running for 00:00:26.29)
Memory usage on this node: 21.5/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |
+---------------------+----------+-----------------+--------------+------+------+-------------+


(func pid=5008) [1,  4000] loss: 1.041
== Status ==
Current time: 2022-04-10 04:46:00 (running for 00:00:31.33)
Memory usage on this node: 21.6/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |
+---------------------+----------+-----------------+--------------+------+------+-------------+


(func pid=5004) [1,  6000] loss: 0.647
(func pid=5009) [1,  6000] loss: 0.781
== Status ==
Current time: 2022-04-10 04:46:05 (running for 00:00:36.36)
Memory usage on this node: 21.6/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |
|---------------------+----------+-----------------+--------------+------+------+-------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |
+---------------------+----------+-----------------+--------------+------+------+-------------+


Result for DEFAULT_0c10e_00000:
  accuracy: 0.1328
  date: 2022-04-10_04-46-06
  done: false
  experiment_id: 1f1ce6d2da8349da94a7b7c2a34154b0
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 2.241459646701813
  node_ip: 172.17.0.3
  pid: 5006
  should_checkpoint: true
  time_since_restore: 35.506600856781006
  time_this_iter_s: 35.506600856781006
  time_total_s: 35.506600856781006
  timestamp: 1649565966
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00000

Result for DEFAULT_0c10e_00002:
  accuracy: 0.2314
  date: 2022-04-10_04-46-06
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 1.9600008015632628
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 34.58101677894592
  time_this_iter_s: 34.58101677894592
  time_total_s: 34.58101677894592
  timestamp: 1649565966
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00002

== Status ==
Current time: 2022-04-10 04:46:10 (running for 00:00:41.38)
Memory usage on this node: 21.6/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 2.24146 |     0.1328 |                    1 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |         |            |                      |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.96    |     0.2314 |                    1 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5004) [1,  8000] loss: 0.481
(func pid=5009) [1,  8000] loss: 0.586
== Status ==
Current time: 2022-04-10 04:46:15 (running for 00:00:46.40)
Memory usage on this node: 21.6/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 2.24146 |     0.1328 |                    1 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |         |            |                      |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.96    |     0.2314 |                    1 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5006) [2,  2000] loss: 2.176
(func pid=5008) [2,  2000] loss: 1.881
(func pid=5004) [1, 10000] loss: 0.379
== Status ==
Current time: 2022-04-10 04:46:20 (running for 00:00:51.43)
Memory usage on this node: 21.7/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 2.24146 |     0.1328 |                    1 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |         |            |                      |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.96    |     0.2314 |                    1 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5009) [1, 10000] loss: 0.469
== Status ==
Current time: 2022-04-10 04:46:26 (running for 00:00:56.46)
Memory usage on this node: 21.7/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 2.24146 |     0.1328 |                    1 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |         |            |                      |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.96    |     0.2314 |                    1 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5008) [2,  4000] loss: 0.883
(func pid=5006) [2,  4000] loss: 1.010
(func pid=5004) [1, 12000] loss: 0.318
== Status ==
Current time: 2022-04-10 04:46:31 (running for 00:01:01.49)
Memory usage on this node: 21.7/31.3 GiB
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (6 PENDING, 4 RUNNING)
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status   | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING  | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 2.24146 |     0.1328 |                    1 |
| DEFAULT_0c10e_00001 | RUNNING  | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    |         |            |                      |
| DEFAULT_0c10e_00002 | RUNNING  | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.96    |     0.2314 |                    1 |
| DEFAULT_0c10e_00003 | RUNNING  | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | PENDING  |                 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | PENDING  |                 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING  |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING  |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING  |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING  |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
+---------------------+----------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00001:
  accuracy: 0.1014
  date: 2022-04-10_04-46-31
  done: true
  experiment_id: 03200849f7084895a8375703be778bbf
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 2.3629231911182402
  node_ip: 172.17.0.3
  pid: 5009
  should_checkpoint: true
  time_since_restore: 59.77773141860962
  time_this_iter_s: 59.77773141860962
  time_total_s: 59.77773141860962
  timestamp: 1649565991
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00001

(func pid=5005) Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:46:36 (running for 00:01:07.30)
Memory usage on this node: 21.3/31.3 GiB
Using AsyncHyperBand: num_stopped=1
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (5 PENDING, 4 RUNNING, 1 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | RUNNING    | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 2.24146 |     0.1328 |                    1 |
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.96    |     0.2314 |                    1 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | RUNNING    | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | PENDING    |                 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING    |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5005) Files already downloaded and verified
Result for DEFAULT_0c10e_00002:
  accuracy: 0.3859
  date: 2022-04-10_04-46-38
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 2
  loss: 1.645931950187683
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 66.25421857833862
  time_this_iter_s: 31.6732017993927
  time_total_s: 66.25421857833862
  timestamp: 1649565998
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 0c10e_00002

(func pid=5004) [1, 14000] loss: 0.271
Result for DEFAULT_0c10e_00000:
  accuracy: 0.2949
  date: 2022-04-10_04-46-39
  done: true
  experiment_id: 1f1ce6d2da8349da94a7b7c2a34154b0
  hostname: d52bd7357ae4
  iterations_since_restore: 2
  loss: 1.9218169279098511
  node_ip: 172.17.0.3
  pid: 5006
  should_checkpoint: true
  time_since_restore: 68.59252429008484
  time_this_iter_s: 33.08592343330383
  time_total_s: 68.59252429008484
  timestamp: 1649565999
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 0c10e_00000

== Status ==
Current time: 2022-04-10 04:46:42 (running for 00:01:12.85)
Memory usage on this node: 21.2/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (5 PENDING, 3 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.64593 |     0.3859 |                    2 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | RUNNING    | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | PENDING    |                 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING    |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5003) Files already downloaded and verified
(func pid=5003) Files already downloaded and verified
(func pid=5004) [1, 16000] loss: 0.237
== Status ==
Current time: 2022-04-10 04:46:47 (running for 00:01:17.88)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.64593 |     0.3859 |                    2 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | RUNNING    | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING    |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5008) [3,  2000] loss: 1.618
(func pid=5005) [1,  2000] loss: 2.302
== Status ==
Current time: 2022-04-10 04:46:52 (running for 00:01:22.93)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.64593 |     0.3859 |                    2 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | RUNNING    | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING    |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5004) [1, 18000] loss: 0.211
== Status ==
Current time: 2022-04-10 04:46:57 (running for 00:01:27.95)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=2
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (4 PENDING, 4 RUNNING, 2 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.64593 |     0.3859 |                    2 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00004 | RUNNING    | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 |         |            |                      |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | PENDING    |                 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00004:
  accuracy: 0.1769
  date: 2022-04-10_04-46-58
  done: true
  experiment_id: 57ed9d0ac381450aa6becf0247e0c785
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 2.276254405593872
  node_ip: 172.17.0.3
  pid: 5005
  should_checkpoint: true
  time_since_restore: 22.50801730155945
  time_this_iter_s: 22.50801730155945
  time_total_s: 22.50801730155945
  timestamp: 1649566018
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00004

(func pid=5003) [1,  2000] loss: 2.225
(func pid=5008) [3,  4000] loss: 0.773
== Status ==
Current time: 2022-04-10 04:47:03 (running for 00:01:33.46)
Memory usage on this node: 20.3/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.2588570261478425
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.64593 |     0.3859 |                    2 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  |         |            |                      |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) Files already downloaded and verified
(func pid=5007) Files already downloaded and verified
Result for DEFAULT_0c10e_00005:
  accuracy: 0.2201
  date: 2022-04-10_04-47-06
  done: false
  experiment_id: f40073fe155e42f98d126de338b9f00b
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 1.9598983837127686
  node_ip: 172.17.0.3
  pid: 5003
  should_checkpoint: true
  time_since_restore: 22.279260635375977
  time_this_iter_s: 22.279260635375977
  time_total_s: 22.279260635375977
  timestamp: 1649566026
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00005

(func pid=5004) [1, 20000] loss: 0.188
== Status ==
Current time: 2022-04-10 04:47:08 (running for 00:01:38.71)
Memory usage on this node: 20.6/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.64593 |     0.3859 |                    2 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.9599  |     0.2201 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00002:
  accuracy: 0.4493
  date: 2022-04-10_04-47-11
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 3
  loss: 1.498528915500641
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 99.243403673172
  time_this_iter_s: 32.989185094833374
  time_total_s: 99.243403673172
  timestamp: 1649566031
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 0c10e_00002

== Status ==
Current time: 2022-04-10 04:47:13 (running for 00:01:44.01)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.49853 |     0.4493 |                    3 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.9599  |     0.2201 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [1,  2000] loss: 2.200
== Status ==
Current time: 2022-04-10 04:47:18 (running for 00:01:49.04)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.49853 |     0.4493 |                    3 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  |         |            |                      |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.9599  |     0.2201 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5003) [2,  2000] loss: 1.809
(func pid=5008) [4,  2000] loss: 1.480
Result for DEFAULT_0c10e_00003:
  accuracy: 0.2992
  date: 2022-04-10_04-47-23
  done: false
  experiment_id: 3c43cf11e6bf473a9355f90923040cdd
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 1.8393557698599994
  node_ip: 172.17.0.3
  pid: 5004
  should_checkpoint: true
  time_since_restore: 111.17262768745422
  time_this_iter_s: 111.17262768745422
  time_total_s: 111.17262768745422
  timestamp: 1649566043
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00003

== Status ==
Current time: 2022-04-10 04:47:24 (running for 00:01:54.73)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.783874439048767 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.49853 |     0.4493 |                    3 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.9599  |     0.2201 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [1,  4000] loss: 1.015
Result for DEFAULT_0c10e_00005:
  accuracy: 0.3477
  date: 2022-04-10_04-47-26
  done: false
  experiment_id: f40073fe155e42f98d126de338b9f00b
  hostname: d52bd7357ae4
  iterations_since_restore: 2
  loss: 1.6899889686584473
  node_ip: 172.17.0.3
  pid: 5003
  should_checkpoint: true
  time_since_restore: 42.91800880432129
  time_this_iter_s: 20.638748168945312
  time_total_s: 42.91800880432129
  timestamp: 1649566046
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 0c10e_00005

== Status ==
Current time: 2022-04-10 04:47:29 (running for 00:02:00.35)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.49853 |     0.4493 |                    3 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.68999 |     0.3477 |                    2 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5004) [2,  2000] loss: 1.893
(func pid=5008) [4,  4000] loss: 0.723
(func pid=5007) [1,  6000] loss: 0.637
== Status ==
Current time: 2022-04-10 04:47:34 (running for 00:02:05.39)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.49853 |     0.4493 |                    3 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.68999 |     0.3477 |                    2 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


== Status ==
Current time: 2022-04-10 04:47:39 (running for 00:02:10.42)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.49853 |     0.4493 |                    3 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.68999 |     0.3477 |                    2 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5003) [3,  2000] loss: 1.663
(func pid=5004) [2,  4000] loss: 0.955
(func pid=5007) [1,  8000] loss: 0.463
Result for DEFAULT_0c10e_00002:
  accuracy: 0.5001
  date: 2022-04-10_04-47-44
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 4
  loss: 1.3877728350639342
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 132.1592869758606
  time_this_iter_s: 32.9158833026886
  time_total_s: 132.1592869758606
  timestamp: 1649566064
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 0c10e_00002

== Status ==
Current time: 2022-04-10 04:47:45 (running for 00:02:15.94)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.38777 |     0.5001 |                    4 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.68999 |     0.3477 |                    2 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00005:
  accuracy: 0.3869
  date: 2022-04-10_04-47-47
  done: false
  experiment_id: f40073fe155e42f98d126de338b9f00b
  hostname: d52bd7357ae4
  iterations_since_restore: 3
  loss: 1.601786145591736
  node_ip: 172.17.0.3
  pid: 5003
  should_checkpoint: true
  time_since_restore: 63.52823877334595
  time_this_iter_s: 20.610229969024658
  time_total_s: 63.52823877334595
  timestamp: 1649566067
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 0c10e_00005

== Status ==
Current time: 2022-04-10 04:47:50 (running for 00:02:20.96)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.38777 |     0.5001 |                    4 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.60179 |     0.3869 |                    3 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5004) [2,  6000] loss: 0.635
(func pid=5007) [1, 10000] loss: 0.358
== Status ==
Current time: 2022-04-10 04:47:55 (running for 00:02:25.99)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.38777 |     0.5001 |                    4 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.60179 |     0.3869 |                    3 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5008) [5,  2000] loss: 1.392
== Status ==
Current time: 2022-04-10 04:48:00 (running for 00:02:31.02)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.38777 |     0.5001 |                    4 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.60179 |     0.3869 |                    3 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5004) [2,  8000] loss: 0.476
(func pid=5003) [4,  2000] loss: 1.581
(func pid=5007) [1, 12000] loss: 0.293
== Status ==
Current time: 2022-04-10 04:48:05 (running for 00:02:36.06)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=3
Bracket: Iter 8.000: None | Iter 4.000: -1.3877728350639342 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 4 RUNNING, 3 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.38777 |     0.5001 |                    4 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00005 | RUNNING    | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.60179 |     0.3869 |                    3 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5008) [5,  4000] loss: 0.680
Result for DEFAULT_0c10e_00005:
  accuracy: 0.393
  date: 2022-04-10_04-48-08
  done: true
  experiment_id: f40073fe155e42f98d126de338b9f00b
  hostname: d52bd7357ae4
  iterations_since_restore: 4
  loss: 1.5446142356872559
  node_ip: 172.17.0.3
  pid: 5003
  should_checkpoint: true
  time_since_restore: 84.13721871376038
  time_this_iter_s: 20.60897994041443
  time_total_s: 84.13721871376038
  timestamp: 1649566088
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 0c10e_00005

(func pid=5004) [2, 10000] loss: 0.380
== Status ==
Current time: 2022-04-10 04:48:11 (running for 00:02:41.57)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 PENDING, 3 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.38777 |     0.5001 |                    4 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | PENDING    |                 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [1, 14000] loss: 0.251
(func pid=5002) Files already downloaded and verified
(func pid=5002) Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:48:16 (running for 00:02:46.62)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.38777 |     0.5001 |                    4 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | RUNNING    | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00002:
  accuracy: 0.5279
  date: 2022-04-10_04-48-17
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 5
  loss: 1.3209753231763839
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 165.1197385787964
  time_this_iter_s: 32.96045160293579
  time_total_s: 165.1197385787964
  timestamp: 1649566097
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 0c10e_00002

(func pid=5004) [2, 12000] loss: 0.325
(func pid=5007) [1, 16000] loss: 0.219
== Status ==
Current time: 2022-04-10 04:48:21 (running for 00:02:51.89)
Memory usage on this node: 20.7/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.32098 |     0.5279 |                    5 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | RUNNING    | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


== Status ==
Current time: 2022-04-10 04:48:26 (running for 00:02:56.93)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.32098 |     0.5279 |                    5 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | RUNNING    | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5002) [1,  2000] loss: 2.294
(func pid=5004) [2, 14000] loss: 0.278
(func pid=5008) [6,  2000] loss: 1.306
(func pid=5007) [1, 18000] loss: 0.200
== Status ==
Current time: 2022-04-10 04:48:31 (running for 00:03:01.96)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=4
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 4 RUNNING, 4 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.32098 |     0.5279 |                    5 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00007 | RUNNING    | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00007:
  accuracy: 0.1021
  date: 2022-04-10_04-48-35
  done: true
  experiment_id: 8c02aed1ca0b430f8522294784a6b331
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 2.3056997585296632
  node_ip: 172.17.0.3
  pid: 5002
  should_checkpoint: true
  time_since_restore: 22.65566062927246
  time_this_iter_s: 22.65566062927246
  time_total_s: 22.65566062927246
  timestamp: 1649566115
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00007

== Status ==
Current time: 2022-04-10 04:48:37 (running for 00:03:07.75)
Memory usage on this node: 20.3/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 PENDING, 3 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.32098 |     0.5279 |                    5 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00008 | PENDING    |                 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5004) [2, 16000] loss: 0.239
(func pid=5007) [1, 20000] loss: 0.176
(func pid=5008) [6,  4000] loss: 0.651
(func pid=8344) Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:48:42 (running for 00:03:12.78)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.32098 |     0.5279 |                    5 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00008 | RUNNING    | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=8344) Files already downloaded and verified
(func pid=5004) [2, 18000] loss: 0.215
== Status ==
Current time: 2022-04-10 04:48:47 (running for 00:03:17.83)
Memory usage on this node: 20.9/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.32098 |     0.5279 |                    5 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00008 | RUNNING    | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00002:
  accuracy: 0.5379
  date: 2022-04-10_04-48-50
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 6
  loss: 1.279394944357872
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 197.71211290359497
  time_this_iter_s: 32.592374324798584
  time_total_s: 197.71211290359497
  timestamp: 1649566130
  timesteps_since_restore: 0
  training_iteration: 6
  trial_id: 0c10e_00002

== Status ==
Current time: 2022-04-10 04:48:53 (running for 00:03:23.49)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.27939 |     0.5379 |                    6 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  |         |            |                      |
| DEFAULT_0c10e_00008 | RUNNING    | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00006:
  accuracy: 0.3431
  date: 2022-04-10_04-48-55
  done: false
  experiment_id: 59808294236046a390077b9dc120b8cf
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 1.7145750304013492
  node_ip: 172.17.0.3
  pid: 5007
  should_checkpoint: true
  time_since_restore: 111.48537373542786
  time_this_iter_s: 111.48537373542786
  time_total_s: 111.48537373542786
  timestamp: 1649566135
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00006

(func pid=5004) [2, 20000] loss: 0.197
(func pid=8344) [1,  2000] loss: 2.313
== Status ==
Current time: 2022-04-10 04:48:58 (running for 00:03:28.59)
Memory usage on this node: 20.8/31.3 GiB
Using AsyncHyperBand: num_stopped=5
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.100730224132538
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 4 RUNNING, 5 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.27939 |     0.5379 |                    6 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00008 | RUNNING    | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   |         |            |                      |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5008) [7,  2000] loss: 1.246
Result for DEFAULT_0c10e_00008:
  accuracy: 0.1007
  date: 2022-04-10_04-49-02
  done: true
  experiment_id: 1b12f08297d74f769c4bf4aca1a85bce
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 2.3078824977874755
  node_ip: 172.17.0.3
  pid: 8344
  should_checkpoint: true
  time_since_restore: 22.114116430282593
  time_this_iter_s: 22.114116430282593
  time_total_s: 22.114116430282593
  timestamp: 1649566142
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00008

== Status ==
Current time: 2022-04-10 04:49:04 (running for 00:03:34.46)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 PENDING, 3 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.27939 |     0.5379 |                    6 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | PENDING    |                 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [2,  2000] loss: 1.713
(func pid=9009) Files already downloaded and verified
== Status ==
Current time: 2022-04-10 04:49:09 (running for 00:03:39.50)
Memory usage on this node: 20.4/31.3 GiB
Using AsyncHyperBand: num_stopped=6
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.6899889686584473 | Iter 1.000: -2.241459646701813
Resources requested: 8.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (4 RUNNING, 6 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.27939 |     0.5379 |                    6 |
| DEFAULT_0c10e_00003 | RUNNING    | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 1.83936 |     0.2992 |                    1 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=9009) Files already downloaded and verified
Result for DEFAULT_0c10e_00003:
  accuracy: 0.2373
  date: 2022-04-10_04-49-10
  done: true
  experiment_id: 3c43cf11e6bf473a9355f90923040cdd
  hostname: d52bd7357ae4
  iterations_since_restore: 2
  loss: 2.0048313949137926
  node_ip: 172.17.0.3
  pid: 5004
  should_checkpoint: true
  time_since_restore: 218.33158326148987
  time_this_iter_s: 107.15895557403564
  time_total_s: 218.33158326148987
  timestamp: 1649566150
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 0c10e_00003

(func pid=5008) [7,  4000] loss: 0.628
(func pid=5007) [2,  4000] loss: 0.865
== Status ==
Current time: 2022-04-10 04:49:14 (running for 00:03:44.88)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.27939 |     0.5379 |                    6 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [2,  6000] loss: 0.576
== Status ==
Current time: 2022-04-10 04:49:19 (running for 00:03:49.90)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.27939 |     0.5379 |                    6 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00002:
  accuracy: 0.5528
  date: 2022-04-10_04-49-19
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 7
  loss: 1.2448112061977386
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 227.53409790992737
  time_this_iter_s: 29.821985006332397
  time_total_s: 227.53409790992737
  timestamp: 1649566159
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 0c10e_00002

(func pid=9009) [1,  2000] loss: 2.344
== Status ==
Current time: 2022-04-10 04:49:24 (running for 00:03:55.30)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.24481 |     0.5528 |                    7 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [2,  8000] loss: 0.428
(func pid=5008) [8,  2000] loss: 1.209
== Status ==
Current time: 2022-04-10 04:49:29 (running for 00:04:00.31)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.24481 |     0.5528 |                    7 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=9009) [1,  4000] loss: 1.168
(func pid=5007) [2, 10000] loss: 0.347
== Status ==
Current time: 2022-04-10 04:49:34 (running for 00:04:05.33)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.24481 |     0.5528 |                    7 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5008) [8,  4000] loss: 0.603
(func pid=5007) [2, 12000] loss: 0.291
== Status ==
Current time: 2022-04-10 04:49:39 (running for 00:04:10.34)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.24481 |     0.5528 |                    7 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=9009) [1,  6000] loss: 0.780
== Status ==
Current time: 2022-04-10 04:49:44 (running for 00:04:15.35)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: None | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.24481 |     0.5528 |                    7 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00002:
  accuracy: 0.5533
  date: 2022-04-10_04-49-46
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 8
  loss: 1.2396061965227128
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 253.7500820159912
  time_this_iter_s: 26.215984106063843
  time_total_s: 253.7500820159912
  timestamp: 1649566186
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 0c10e_00002

(func pid=5007) [2, 14000] loss: 0.249
== Status ==
Current time: 2022-04-10 04:49:50 (running for 00:04:20.52)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.23961 |     0.5533 |                    8 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [2, 16000] loss: 0.220
(func pid=9009) [1,  8000] loss: 0.585
(func pid=5008) [9,  2000] loss: 1.171
== Status ==
Current time: 2022-04-10 04:49:55 (running for 00:04:25.53)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.23961 |     0.5533 |                    8 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [2, 18000] loss: 0.193
== Status ==
Current time: 2022-04-10 04:50:00 (running for 00:04:30.55)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.23961 |     0.5533 |                    8 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5008) [9,  4000] loss: 0.588
(func pid=9009) [1, 10000] loss: 0.468
== Status ==
Current time: 2022-04-10 04:50:05 (running for 00:04:35.56)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.23961 |     0.5533 |                    8 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [2, 20000] loss: 0.174
== Status ==
Current time: 2022-04-10 04:50:10 (running for 00:04:40.57)
Memory usage on this node: 20.2/31.3 GiB
Using AsyncHyperBand: num_stopped=7
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.241459646701813
Resources requested: 6.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (3 RUNNING, 7 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.23961 |     0.5533 |                    8 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00009 | RUNNING    | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   |         |            |                      |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00009:
  accuracy: 0.1029
  date: 2022-04-10_04-50-11
  done: true
  experiment_id: 8bd00d26b5494a6eb8a6e14f7f4cd522
  hostname: d52bd7357ae4
  iterations_since_restore: 1
  loss: 2.325470637321472
  node_ip: 172.17.0.3
  pid: 9009
  should_checkpoint: true
  time_since_restore: 63.21910643577576
  time_this_iter_s: 63.21910643577576
  time_total_s: 63.21910643577576
  timestamp: 1649566211
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0c10e_00009

Result for DEFAULT_0c10e_00002:
  accuracy: 0.5576
  date: 2022-04-10_04-50-12
  done: false
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 9
  loss: 1.2259806815862655
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 279.8658182621002
  time_this_iter_s: 26.11573624610901
  time_total_s: 279.8658182621002
  timestamp: 1649566212
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 0c10e_00002

== Status ==
Current time: 2022-04-10 04:50:15 (running for 00:04:45.63)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.805902948284149 | Iter 1.000: -2.2588570261478425
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22598 |     0.5576 |                    9 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.71458 |     0.3431 |                    1 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00006:
  accuracy: 0.3272
  date: 2022-04-10_04-50-17
  done: false
  experiment_id: 59808294236046a390077b9dc120b8cf
  hostname: d52bd7357ae4
  iterations_since_restore: 2
  loss: 1.782580977988243
  node_ip: 172.17.0.3
  pid: 5007
  should_checkpoint: true
  time_since_restore: 193.41082763671875
  time_this_iter_s: 81.9254539012909
  time_total_s: 193.41082763671875
  timestamp: 1649566217
  timesteps_since_restore: 0
  training_iteration: 2
  trial_id: 0c10e_00006

(func pid=5008) [10,  2000] loss: 1.144
== Status ==
Current time: 2022-04-10 04:50:21 (running for 00:04:51.51)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22598 |     0.5576 |                    9 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3,  2000] loss: 1.718
== Status ==
Current time: 2022-04-10 04:50:26 (running for 00:04:56.52)
Memory usage on this node: 19.8/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22598 |     0.5576 |                    9 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5008) [10,  4000] loss: 0.579
(func pid=5007) [3,  4000] loss: 0.853
== Status ==
Current time: 2022-04-10 04:50:31 (running for 00:05:01.53)
Memory usage on this node: 19.8/31.3 GiB
Using AsyncHyperBand: num_stopped=8
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 4.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (2 RUNNING, 8 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00002 | RUNNING    | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22598 |     0.5576 |                    9 |
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3,  6000] loss: 0.583
Result for DEFAULT_0c10e_00002:
  accuracy: 0.5627
  date: 2022-04-10_04-50-35
  done: true
  experiment_id: fd74a8193f3640b1be44dc88dd569b6a
  hostname: d52bd7357ae4
  iterations_since_restore: 10
  loss: 1.222454554271698
  node_ip: 172.17.0.3
  pid: 5008
  should_checkpoint: true
  time_since_restore: 303.0206735134125
  time_this_iter_s: 23.154855251312256
  time_total_s: 303.0206735134125
  timestamp: 1649566235
  timesteps_since_restore: 0
  training_iteration: 10
  trial_id: 0c10e_00002

== Status ==
Current time: 2022-04-10 04:50:36 (running for 00:05:06.78)
Memory usage on this node: 19.3/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3,  8000] loss: 0.435
== Status ==
Current time: 2022-04-10 04:50:41 (running for 00:05:11.80)
Memory usage on this node: 19.3/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3, 10000] loss: 0.351
== Status ==
Current time: 2022-04-10 04:50:46 (running for 00:05:16.81)
Memory usage on this node: 19.4/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3, 12000] loss: 0.285
== Status ==
Current time: 2022-04-10 04:50:51 (running for 00:05:21.82)
Memory usage on this node: 19.4/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3, 14000] loss: 0.250
== Status ==
Current time: 2022-04-10 04:50:56 (running for 00:05:26.82)
Memory usage on this node: 19.4/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3, 16000] loss: 0.216
== Status ==
Current time: 2022-04-10 04:51:01 (running for 00:05:31.84)
Memory usage on this node: 19.5/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3, 18000] loss: 0.196
== Status ==
Current time: 2022-04-10 04:51:06 (running for 00:05:36.84)
Memory usage on this node: 19.5/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [3, 20000] loss: 0.170
== Status ==
Current time: 2022-04-10 04:51:11 (running for 00:05:41.86)
Memory usage on this node: 19.5/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.78258 |     0.3272 |                    2 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00006:
  accuracy: 0.3594
  date: 2022-04-10_04-51-15
  done: false
  experiment_id: 59808294236046a390077b9dc120b8cf
  hostname: d52bd7357ae4
  iterations_since_restore: 3
  loss: 1.7754648028180002
  node_ip: 172.17.0.3
  pid: 5007
  should_checkpoint: true
  time_since_restore: 251.42181205749512
  time_this_iter_s: 58.01098442077637
  time_total_s: 251.42181205749512
  timestamp: 1649566275
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 0c10e_00006

== Status ==
Current time: 2022-04-10 04:51:17 (running for 00:05:47.50)
Memory usage on this node: 19.6/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4,  2000] loss: 1.746
== Status ==
Current time: 2022-04-10 04:51:22 (running for 00:05:52.52)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4,  4000] loss: 0.861
== Status ==
Current time: 2022-04-10 04:51:27 (running for 00:05:57.52)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4,  6000] loss: 0.560
== Status ==
Current time: 2022-04-10 04:51:32 (running for 00:06:02.54)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4,  8000] loss: 0.445
== Status ==
Current time: 2022-04-10 04:51:37 (running for 00:06:07.54)
Memory usage on this node: 19.8/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4, 10000] loss: 0.349
== Status ==
Current time: 2022-04-10 04:51:42 (running for 00:06:12.56)
Memory usage on this node: 19.8/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4, 12000] loss: 0.292
== Status ==
Current time: 2022-04-10 04:51:47 (running for 00:06:17.57)
Memory usage on this node: 19.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4, 14000] loss: 0.250
== Status ==
Current time: 2022-04-10 04:51:52 (running for 00:06:22.58)
Memory usage on this node: 19.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4, 16000] loss: 0.217
(func pid=5007) [4, 18000] loss: 0.193
== Status ==
Current time: 2022-04-10 04:51:57 (running for 00:06:27.59)
Memory usage on this node: 19.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


(func pid=5007) [4, 20000] loss: 0.171
== Status ==
Current time: 2022-04-10 04:52:02 (running for 00:06:32.60)
Memory usage on this node: 19.9/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


== Status ==
Current time: 2022-04-10 04:52:07 (running for 00:06:37.61)
Memory usage on this node: 20.0/31.3 GiB
Using AsyncHyperBand: num_stopped=9
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.466193535375595 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 2.0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (1 RUNNING, 9 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00006 | RUNNING    | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.77546 |     0.3594 |                    3 |
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Result for DEFAULT_0c10e_00006:
  accuracy: 0.3782
  date: 2022-04-10_04-52-09
  done: true
  experiment_id: 59808294236046a390077b9dc120b8cf
  hostname: d52bd7357ae4
  iterations_since_restore: 4
  loss: 1.7262782471820712
  node_ip: 172.17.0.3
  pid: 5007
  should_checkpoint: true
  time_since_restore: 305.57490062713623
  time_this_iter_s: 54.15308856964111
  time_total_s: 305.57490062713623
  timestamp: 1649566329
  timesteps_since_restore: 0
  training_iteration: 4
  trial_id: 0c10e_00006

== Status ==
Current time: 2022-04-10 04:52:09 (running for 00:06:39.66)
Memory usage on this node: 19.7/31.3 GiB
Using AsyncHyperBand: num_stopped=10
Bracket: Iter 8.000: -1.2396061965227128 | Iter 4.000: -1.5446142356872559 | Iter 2.000: -1.782580977988243 | Iter 1.000: -2.2588570261478425
Resources requested: 0/8 CPUs, 0/2 GPUs, 0.0/11.68 GiB heap, 0.0/5.84 GiB objects (0.0/1.0 accelerator_type:GTX)
Result logdir: /root/ray_results/DEFAULT_2022-04-10_04-45-29
Number of trials: 10/10 (10 TERMINATED)
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+
| Trial name          | status     | loc             |   batch_size |   l1 |   l2 |          lr |    loss |   accuracy |   training_iteration |
|---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------|
| DEFAULT_0c10e_00000 | TERMINATED | 172.17.0.3:5006 |            8 |  128 |   32 | 0.000234003 | 1.92182 |     0.2949 |                    2 |
| DEFAULT_0c10e_00001 | TERMINATED | 172.17.0.3:5009 |            4 |    4 |    4 | 0.073943    | 2.36292 |     0.1014 |                    1 |
| DEFAULT_0c10e_00002 | TERMINATED | 172.17.0.3:5008 |            8 |   32 |    8 | 0.000505412 | 1.22245 |     0.5627 |                   10 |
| DEFAULT_0c10e_00003 | TERMINATED | 172.17.0.3:5004 |            2 |   16 |   16 | 0.00365822  | 2.00483 |     0.2373 |                    2 |
| DEFAULT_0c10e_00004 | TERMINATED | 172.17.0.3:5005 |           16 |    4 |  128 | 0.000156333 | 2.27625 |     0.1769 |                    1 |
| DEFAULT_0c10e_00005 | TERMINATED | 172.17.0.3:5003 |           16 |    8 |    4 | 0.00147991  | 1.54461 |     0.393  |                    4 |
| DEFAULT_0c10e_00006 | TERMINATED | 172.17.0.3:5007 |            2 |    8 |    8 | 0.00251674  | 1.72628 |     0.3782 |                    4 |
| DEFAULT_0c10e_00007 | TERMINATED | 172.17.0.3:5002 |           16 |    8 |   16 | 0.0569105   | 2.3057  |     0.1021 |                    1 |
| DEFAULT_0c10e_00008 | TERMINATED | 172.17.0.3:8344 |           16 |   16 |   16 | 0.0642732   | 2.30788 |     0.1007 |                    1 |
| DEFAULT_0c10e_00009 | TERMINATED | 172.17.0.3:9009 |            4 |  256 |   64 | 0.0648686   | 2.32547 |     0.1029 |                    1 |
+---------------------+------------+-----------------+--------------+------+------+-------------+---------+------------+----------------------+


Best trial config: {'l1': 32, 'l2': 8, 'lr': 0.0005054115351056523, 'batch_size': 8}
Best trial final validation loss: 1.222454554271698
Best trial final validation accuracy: 0.5627
Files already downloaded and verified
Files already downloaded and verified
Best trial test set accuracy: 0.5672

If you run the code, an example output could look like this:

Number of trials: 10 (10 TERMINATED)
+-----+------+------+-------------+--------------+---------+------------+--------------------+
| ... |   l1 |   l2 |          lr |   batch_size |    loss |   accuracy | training_iteration |
|-----+------+------+-------------+--------------+---------+------------+--------------------|
| ... |   64 |    4 | 0.00011629  |            2 | 1.87273 |     0.244  |                  2 |
| ... |   32 |   64 | 0.000339763 |            8 | 1.23603 |     0.567  |                  8 |
| ... |    8 |   16 | 0.00276249  |           16 | 1.1815  |     0.5836 |                 10 |
| ... |    4 |   64 | 0.000648721 |            4 | 1.31131 |     0.5224 |                  8 |
| ... |   32 |   16 | 0.000340753 |            8 | 1.26454 |     0.5444 |                  8 |
| ... |    8 |    4 | 0.000699775 |            8 | 1.99594 |     0.1983 |                  2 |
| ... |  256 |    8 | 0.0839654   |           16 | 2.3119  |     0.0993 |                  1 |
| ... |   16 |  128 | 0.0758154   |           16 | 2.33575 |     0.1327 |                  1 |
| ... |   16 |    8 | 0.0763312   |           16 | 2.31129 |     0.1042 |                  4 |
| ... |  128 |   16 | 0.000124903 |            4 | 2.26917 |     0.1945 |                  1 |
+-----+------+------+-------------+--------------+---------+------------+--------------------+


Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
Best trial final validation loss: 1.181501
Best trial final validation accuracy: 0.5836
Best trial test set accuracy: 0.5806

Most trials have been stopped early in order to avoid wasting resources. The best performing trial achieved a validation accuracy of about 58%, which could be confirmed on the test set.

So that’s it! You can now tune the parameters of your PyTorch models.