Dolly gpt training does not record model #53273

Marcos-gi · 2023-04-20T03:51:48Z

Marcos-gi
Apr 20, 2023

Select Topic Area

Bug

Body

Hi,
I was able to run the suggested workout on a jupyter anaconda on windows. However, the model was not recorded.
How do I solve it and what's the next step?

import torch 
import logging
import dbutils as dbutils
import os
from datetime import datetime
from training.trainer import load_training_dataset, load_tokenizer
logging.basicConfig(
    format="%(asctime)s %(levelname)s [%(name)s] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S"
)
logging.getLogger("py4j").setLevel(logging.WARNING)
logging.getLogger("sh.command").setLevel(logging.ERROR)
load_training_dataset()
load_tokenizer()
timestamp = datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
model_name = "dolly"
checkpoint_dir_name = f"{model_name}__{timestamp}"
root_path = os.getcwd()
deepspeed_config = os.path.join(root_path, "config\ds_z3_bf16_config.json")
local_training_root = os.path.join(os.path.expanduser('~'), "dolly_training")
os.makedirs(local_training_root, exist_ok=True)
local_output_dir = os.path.join(local_training_root, checkpoint_dir_name)
dbfs_output_dir  = os.path.join(local_training_root, "\dbfs" , checkpoint_dir_name )
num_gpus_flag = ""
num_gpus = 700 #dbutils.widgets.get("num_gpus")
if num_gpus:
    num_gpus = int(num_gpus)
    num_gpus_flag = f"--num_gpus={num_gpus}"
tensorboard_display_dir = f"{local_output_dir}\runs"
print(f"Local Output Dir: {local_output_dir}")
print(f"DBFS Output Dir: {dbfs_output_dir}")
print(f"Tensorboard Display Dir: {tensorboard_display_dir}")

Local Output Dir: C:\Users\Marcos\dolly_training\dolly__2023-04-20T00:33:37
DBFS Output Dir: \dbfs\dolly__2023-04-20T00:33:37
unssorboard Display Dir: C:\Users\Marcos\dolly_training\dolly__2023-04-20T00:33:37

queenofcorgis · 2023-04-20T13:46:06Z

queenofcorgis
Apr 20, 2023
Maintainer

Thanks for posting in the GitHub Community, @Marcos-gi !

We’ve moved your post to our new Machine Learning and MLOps 🧠 category, which is more appropriate for this type of discussion. Best of luck!

0 replies

sethjuarez · 2023-04-24T18:21:27Z

sethjuarez
Apr 24, 2023

Hi Marcos!
Looking at the code I'm not seeing where the model is actually produced (i.e. I don't see the model training being kicked off). Is there more code that might be missing?

2 replies

Marcos-gi May 3, 2023
Author

Desculpe Juarez, Estava viajando.

trainer python.txt

Marcos-gi May 3, 2023
Author

consegui avançar, incluindo a função train no código. Agora está caindo por timeout quando baixa o arquivo pytorch_model.bin. Já tenho ele baixado. Gostaria de saber como fazer o programa reconhecer que ele já está baixado e continuar a execução, sem baixa-lo novamente.

Marcos-gi · 2023-05-03T02:45:55Z

Marcos-gi
May 3, 2023
Author

`import logging

import dbutils as dbutils

logging.basicConfig(
format="%(asctime)s %(levelname)s [%(name)s] %(message)s", level=logging.INFO, datefmt="%Y-%m-%d %H:%M:%S"
)
logging.getLogger("py4j").setLevel(logging.WARNING)
logging.getLogger("sh.command").setLevel(logging.ERROR)

import os
from datetime import datetime
from training.trainer import load_training_dataset, load_tokenizer, train

#dbutils.widgets.text("num_gpus", "", "num_gpus")

load_training_dataset()
load_tokenizer()

timestamp = datetime.now().strftime("%Y-%m-%dT%H:%M:%S")
model_name = "dolly"
checkpoint_dir_name = f"{model_name}__{timestamp}"

root_path = os.getcwd()
deepspeed_config = os.path.join(root_path, "config\ds_z3_bf16_config.json")

local_training_root = os.path.join(os.path.expanduser(''), "dolly_training")
local_training_dbfs = os.path.join(os.path.expanduser(''), "dolly_dbfs")

os.makedirs(local_training_root, exist_ok=True)
os.makedirs(local_training_dbfs, exist_ok=True)

local_output_dir = os.path.join(local_training_root, checkpoint_dir_name)
#dbfs_output_dir = os.path.join("\dbfs\dolly_training", checkpoint_dir_name)
dbfs_output_dir = os.path.join(local_training_dbfs, checkpoint_dir_name)

num_gpus_flag = ""
num_gpus = 700 #dbutils.widgets.get("num_gpus")
if num_gpus:
num_gpus = int(num_gpus)
num_gpus_flag = f"--num_gpus={num_gpus}"

#tensorboard_display_dir = f"{local_output_dir}\runs"
tensorboard_display_dir = f"{local_output_dir}\RUNS"

print(f"Local Output Dir: {local_output_dir}")
print(f"DBFS Output Dir: {dbfs_output_dir}")
print(f"Tensorboard Display Dir: {tensorboard_display_dir}")

os.environ["TOKENIZERS_PARALLELISM"] = "false"

train(
local_output_dir,
dbfs_output_dir,
1,
8,
8,
1e-5,
42,
None,
False,
1,
True,
test_size = 1000)`

1 reply

Marcos-gi May 3, 2023
Author

D:\GPT\TESTDOLLY\venv\Scripts\python.exe D:\GPT\TESTDOLLY\train_dolly.py
2023-05-02 23:50:11 INFO [training.trainer] Loading tatsu-lab/alpaca dataset
2023-05-02 23:50:15 WARNING [datasets.builder] Using custom data configuration tatsu-lab--alpaca-715f206eec35a791
2023-05-02 23:50:15 WARNING [datasets.builder] Found cached dataset parquet (C:/Users/Marcos/.cache/huggingface/datasets/tatsu-lab___parquet/tatsu-lab--alpaca-715f206eec35a791/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)
100%|██████████| 1/1 [00:00<00:00, 200.06it/s]
2023-05-02 23:50:15 INFO [training.trainer] Found 52002 rows
2023-05-02 23:50:15 WARNING [datasets.arrow_dataset] Loading cached processed dataset at C:\Users\Marcos.cache\huggingface\datasets\tatsu-lab___parquet\tatsu-lab--alpaca-715f206eec35a791\0.0.0\2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec\cache-35171739a2faf17f.arrow
2023-05-02 23:50:15 WARNING [datasets.arrow_dataset] Loading cached processed dataset at C:\Users\Marcos.cache\huggingface\datasets\tatsu-lab___parquet\tatsu-lab--alpaca-715f206eec35a791\0.0.0\2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec\cache-e15074070625d15b.arrow
2023-05-02 23:50:15 INFO [training.trainer] Loading tokenizer for EleutherAI/gpt-j-6B
Local Output Dir: C:\Users\Marcos\dolly_training\dolly__2023-05-02T23:50:16
DBFS Output Dir: C:\Users\Marcos\dolly_dbfs\dolly__2023-05-02T23:50:16
Tensorboard Display Dir: C:\Users\Marcos\dolly_training\dolly__2023-05-02T23:50:16\RUNS
2023-05-02 23:50:16 INFO [training.trainer] Loading tokenizer for EleutherAI/gpt-j-6B
2023-05-02 23:50:17 INFO [training.trainer] Loading model for EleutherAI/gpt-j-6B
Downloading pytorch_model.bin: 0%| | 62.9M/24.2G [00:08<52:48, 7.62MB/s]Traceback (most recent call last):

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Community

Dolly gpt training does not record model #53273

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

GitHub Community

Dolly gpt training does not record model #53273

Marcos-gi Apr 20, 2023

Select Topic Area

Body

Replies: 3 comments · 3 replies

queenofcorgis Apr 20, 2023 Maintainer

sethjuarez Apr 24, 2023

Marcos-gi May 3, 2023 Author

Marcos-gi May 3, 2023 Author

Marcos-gi May 3, 2023 Author

Marcos-gi May 3, 2023 Author

Marcos-gi
Apr 20, 2023

Replies: 3 comments 3 replies

queenofcorgis
Apr 20, 2023
Maintainer

sethjuarez
Apr 24, 2023

Marcos-gi May 3, 2023
Author

Marcos-gi May 3, 2023
Author

Marcos-gi
May 3, 2023
Author

Marcos-gi May 3, 2023
Author