Views
No views yet
| Metric | Score |
|---|---|
| Exact Match | 13.67% |
| Semantic Match (cosine ≥ 0.8) | 60.33% |
| Avg Similarity | 0.776 |
Evaluated on 300 held-out test examples from NL2SH-ALFA. Semantic similarity is computed usingall-MiniLM-L6-v2embeddings and is a better indicator of real-world quality than exact match alone, since multiple Bash commands can be functionally equivalent.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained("dhwanichande29/nl-to-bash")
4tokenizer = AutoTokenizer.from_pretrained("dhwanichande29/nl-to-bash")
5
6system_prompt = "Your task is to translate a natural language instruction to a Bash command. You will receive an instruction in English and output a Bash command that can be run in a Linux terminal."
7
8def translate(instruction):
9 messages = [
10 {"role": "system", "content": system_prompt},
11 {"role": "user", "content": instruction}
12 ]
13 formatted = tokenizer.apply_chat_template(
14 messages,
15 tokenize=False,
16 add_generation_prompt=True
17 )
18 inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
19 outputs = model.generate(**inputs, max_new_tokens=100, do_sample=False)
20 response = outputs[0][inputs.input_ids.shape[-1]:]
21 return tokenizer.decode(response, skip_special_tokens=True).strip()
22
23print(translate("list all files in current directory"))
24# find . -type f| Natural Language | Generated Bash |
|---|---|
| list all files in current directory | find . -type f |
| find all python files | find . -name "*.py" |
| count lines in a text file | wc -l path/to/file |
| remove all .tmp files | find . -name "*.tmp" -exec rm {} \; |
| show disk usage | du -h / |
nl2sh project)