Instructions to use ise-uiuc/Magicoder-S-DS-6.7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ise-uiuc/Magicoder-S-DS-6.7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ise-uiuc/Magicoder-S-DS-6.7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ise-uiuc/Magicoder-S-DS-6.7B")
model = AutoModelForCausalLM.from_pretrained("ise-uiuc/Magicoder-S-DS-6.7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ise-uiuc/Magicoder-S-DS-6.7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ise-uiuc/Magicoder-S-DS-6.7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ise-uiuc/Magicoder-S-DS-6.7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ise-uiuc/Magicoder-S-DS-6.7B

SGLang

How to use ise-uiuc/Magicoder-S-DS-6.7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ise-uiuc/Magicoder-S-DS-6.7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ise-uiuc/Magicoder-S-DS-6.7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ise-uiuc/Magicoder-S-DS-6.7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ise-uiuc/Magicoder-S-DS-6.7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ise-uiuc/Magicoder-S-DS-6.7B with Docker Model Runner:
```
docker model run hf.co/ise-uiuc/Magicoder-S-DS-6.7B
```

What is DS, CL, S-DS and S-CL

#10

by supercharge19 - opened Jan 30, 2024

Discussion

supercharge19

Jan 30, 2024

What are these things and how are models affected by them, which suits best for function calling with what prompt?

dungcta

Mar 12, 2024

As refer to their paper, I think that DS means that model is fined tune from DeepSeek base model, and CL is fined tune from CodeLLama base model

LuMatic

Mar 25, 2024

yeah, CL and DS are quite clear. But what "S-" stand for? ;)

If I understand this tweet (xeet?) correctly, it's a Model additionally trained on Magicoder-Evol-Instruct-110K data set?

So:

Magicoder-CL-7B = CodeLlama 7b fine-tuned on Magicoder-OSS-Instruct-75K data set
Magicoder-S-CL-7B = CodeLlama7b fine-tuned on ise-uiuc/Magicoder-OSS-Instruct-75K and Magicoder-Evol-Instruct-110K data sets?
Magicoder-CL-7B = DeepSeek 7b fine-tuned on Magicoder-OSS-Instruct-75K data set
Magicoder-S-CL-7B = DeepSeek fine-tuned on ise-uiuc/Magicoder-OSS-Instruct-75K and Magicoder-Evol-Instruct-110K data sets?

Is that correct? I may add this to Docs if you want.

Can you also confirm the minimum V-RAM required to run specific models?

I tried to load the Gradio demo but got CUDA out of memory. Tried to allocate 32.00 MiB.- so it requires much more than I was expecting :D

LuMatic

Mar 25, 2024

•

edited Mar 25, 2024

Sorry, my bad... It's already in README... :P
https://github.com/ise-uiuc/magicoder/blob/main/README.md#-dataset

Magicoder-OSS-Instruct-75K: generated through OSS-Instruct using gpt-3.5-turbo-1106 and used to train both Magicoder and Magicoder-S series.
Magicoder-Evol-Instruct-110K: decontaminated and redistributed from theblackcat102/evol-codealpaca-v1, used to further finetune Magicoder series and obtain Magicoder-S models.

But I would add it to the Models table for total clarity (Base Model, Data Sets, and minimum V-RAM required).

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment