Add Gemma phone deployment notebook

[glee2429]

Hey @unslothai, I updated the notebook to use Gemma3 for phone deployment. A main difference in model export is using Optimum (https://github.com/huggingface/optimum-executorch/blob/main/optimum/exporters/executorch/README.md.) Unfortunately I wasn't able to reduce the loss below 17, so I cut the training short.

[gemini-code-assist]

Summary of Changes

Hello @glee2429, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive Jupyter notebook that guides users through the process of preparing, training, and deploying the Gemma3 language model for mobile phone environments. It leverages Unsloth for efficient fine-tuning and integrates with Optimum Executorch for model export, demonstrating a practical workflow for on-device AI. The notebook also addresses specific considerations like Quantization Aware Training and data formatting for Gemma3.

Highlights

• New Notebook for Gemma3 Phone Deployment: A new Jupyter notebook has been added to demonstrate the process of deploying the Gemma3 model on mobile phones.
• Optimum Executorch Integration: The notebook highlights the use of Optimum Executorch for exporting the trained Gemma3 model, a key component for phone deployment.
• Quantization Aware Training (QAT): The model loading step explicitly uses qat_scheme="phone-deployment" to apply QAT, aiming to mitigate quantization degradation for mobile inference.
• Data Preparation for Gemma3 Chat Format: The notebook includes steps for data preparation, utilizing the Gemma-3 chat template and the FineTome-100k dataset, with a focus on standardizing data formats.
• Training Loss Observation: The author noted that the training loss did not drop below 17 during the initial short training run, leading to the decision to cut the training short.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds a new Jupyter notebook for deploying a Gemma3 model on a phone. The notebook demonstrates fine-tuning and exporting the model using Unsloth and Optimum Executorch. My review focuses on improving the correctness and clarity of the notebook. I've identified an incorrect parameter in the model export command which could cause issues, suggested removing unused code, and recommended renaming variables for better readability. Overall, it's a great addition, and with these changes, it will be even more robust and easier to follow.

[Datta0]

Hey @glee2429 I tried running your notebook. It runs fine. The export too works fine.
But 270M model results in 1.7GB file. 270M at 32bit itself would at max take 1.08GiB.
Can you please tell me what all the file contains?

[glee2429]

Hey @Datta0! Thanks for the catch.

Previously, the export script didn't compress the embeddings or weights. This was causing the file size balloon because the Gemma 3 model has a massive vocabulary size of 256k tokens, which is nearly 10x larger than comparable models like Llama 3 (128k) or Phi (32k). This results in a huge embedding matrix that takes up ~1GB on its own in fp32.

In the newest code change, I used the --qlinear 8da4w and --qembedding 8w flags in Optimum to handle this. The final size is now roughly 300MB, which is much more suitable for mobile deployment.

Here is the updated cell:

# Export directly from your local folder using Optimum
# We use --qlinear 8da4w and --qembedding 8w to reduce size
# 8da4w = Int8 dynamic activations + Int4 weights (Fastest on mobile)

!optimum-cli export executorch \
    --model "gemma_phone_model" \
    --task "text-generation" \
    --recipe "xnnpack" \
    --output_dir "gemma_output" \
    --qlinear 8da4w \
    --qembedding 8w

(CC: @danielhanchen)

[glee2429]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds a new Jupyter notebook for demonstrating phone deployment with Gemma3. The notebook is well-structured and provides a clear walkthrough from data preparation to model export. I've found a critical bug in one of the code cells that would prevent it from running, and a minor issue with some dead code. My suggestions address these points to ensure the notebook runs correctly and is maintainable.

[glee2429]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds a new Jupyter notebook demonstrating how to finetune a Gemma-3 model and deploy it to a phone using Unsloth and Executorch. The notebook is a great example, but I've found a few issues. The main concern is with the package installation steps, which use versions that don't seem to be publicly available, making the notebook difficult to reproduce. I've also noticed that the training loss is not converging, and I've suggested a potential fix. Additionally, there are some minor issues with variable naming, incorrect memory usage calculation, and a few missing outputs and typos in the markdown cells. Addressing these points will significantly improve the quality and usability of the notebook.