VERSION: PS2026425:1500

PROCEDURE: Jetson Nano Orin Super setup/configuration

#
# Peter Soroka   

 
###################################

STEP: Required hardware

###################################

 __1.) Nvidia Jetsin Orin Nano Devloper Kit $250
__3.) SanDisk Ultra MicroSD card >= 64G $25
 __4.) M.2 2280 NVME SSD 1TB Storage PCIE Gen3 is fine (since that is the speed of the Nano)about $100
 __5.) Monitor with a DisplayPort, displayport is like an HDMI with one edge of the connector
       squared off, it has a higher bandwidth. You will need a DisplayPort to DisplayPort cable
       the monitor or an HDMI to DisplayPort converter if your monitor only supports HDMI
 __6.) a USB keyboard and Mouse
 __7.) Network connection wired is perfered but wireless will also work


###################################

STEP: Copy the OS to a SD Card

###################################
 __1) Use a SanDisk Ultra MicroSD 64gig Class 10/U1/A1 or faster (Extreme) Note: the
      MicroSD card is just for the first boot we will be converting to a M.2 NVME SSD
      boot for normal operation
      copy


###################################

STEP: Migrate from MicroSD to M.2 NVME SSD boot

###################################

# open a terminal, from the "3 by 3 dots" in the lower left of the screen, use the search "term" to open
 cd # to make sure you are in your home dir
 mkdir -p github # create the dir github in your home dir
 cd github # change to the dir
 git clone https://github.com/jetsonhacks/migrate-jetson-ssd # to download the code from github
 cd migrate-jetson-ssd # change to the dir 
# run the following scripts in order:
 sudo bash make_partitions.sh
 sudo bash copy_partitions.sh
 sudo bash configure_ssd_boot.sh
# shutdown and remove the microSSD and reboot  with just the M.2 NVME SSD installed
# once booted from the M.2 NVME SSD continue from a terminal again
# apt install gparted to increase the size of the partitions on the M.2 NVME SSD
 sudo apt update
 sudo apt install gparted
 gparted # using gparted increase the size of the Apps partition do not use all the disk space we will save some
# for futures updates to other partitions (250G for now will be good)
# use the following to examine the disk space
 df -h
 lsblk 
# verify your M.2 NVME SSD is operating correctly
 lspci
# you should see that the "Non-Volatile memory controller" is on the "0004:01:00.0" bus so enter the following:
 lspci -s 0004:01:00.0
# verify the "LnkSta:" line both "Speed 8GT/s (ok) Width x4 (ok)" should report (ok) 


###################################

STEP: increase the speed/power and make this orin "SUPER"

##################################

# this assumes we have atleast jetpack 6.2.1 loaded
# select "green Nividia Icon" at the uper right of the screen
# select "Power Mode", select "MAXN SUPER"
# to display the selection
 nvpmodel -q
# reboot
 sudo reboot

###################################

STEP: Increase the paging swap space to allow more then the real 8g of mem to be used

###################################
# This will allow for more then the real 8g of mem to be used lets add 24g of swap space

# View what we have now
 free -h
 swapon --show
# Disable the ZRAM (if it is active)
 sudo systemctl disable nvzramconfig
# create the swapfile
 sudo fallocate -l 24G /swapfile
# set premisssions
 sudo chmod 600 /swapfile
# setup swap area
 sudo mkswap /swapfile
# Enable the swap
 sudo swapon /swapfile
# Make it survive a reboot
 cat /etc/fstab # this is what you are starting with
 sudo cp /etc/fstab /etc/fstab.bak # backing it up in case you make a mistake
 sudo echo '/swapfile none swap sw 0 0'|sudo tee -a /etc/fstab # add a line
 cat /etc/fstab # this should be what you started with plus the line added above
# view what we have now
 free -h
 swapon --show

###################################

STEP: verify jtop is working

###################################

 jtop ### if nothing then lets install it
 
# update all the installed packages
 sudo apt update
 sudo apt upgrade 
# install pip (python-install-packager)
 sudo apt install python-pip
# install jetson-stats (jtop)
 sudo pip3 install -U jetson-stats
# reboot
 sudo reboot 
 
# try jtop again
 jtop
# tab over to the "6CTRL"
 set the profiles to "[cool]"
 set the Jetson clocks on boot to "e enable"
 set the NVP modes to "[MAXN SUPER]"
# you can also try to use the following to set them
 sudo nvpmodel -m 2
 sudo jetson_clocks
# to show
 sudo jetson_clocks --show
# reboot to maybe get the clocks started
 sudo reboot
 jtop # check the clocks are working

# fix the message in jtop so the jetpack level is valid (if needed)
# goto the following file "jetson_variables.py usually found at the following
 sudo gedit /usr/local/lib/python3.10/dist-packages/jtop/core/jetson_variables.py
# add in the correct firmware and jetpack level
EXAMPLE:

NVIDIA_JETPACK = {
    # -------- JP6 --------
    "36.4.7": "6.2.1",
    "36.4.2": "6.2",
...



###################################
STEP: Use Flatpak to install the following Apps

###################################
#
 Flatpak is the latest method to get the very latest level of an App onto your system and
 handles the pre-reqs. Go to Flathub.org to get the list of available Apps. If you App is 
 listed but will not install it is most likely that the old level of Ubuntu that the 6.2
 level Jetpak uses or the ARM CPU is not suportted by the FlatPack and another method of
 install will be required.

# Setup for flatpak
 sudo apt update
 sudo apt install -y flatpak
 sudo flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repflathub.flatpakrepo

### MAKER
 sudo flatpak install -y flathub org.gimp.GIMP                 ### GNU Image Manipulation

### PHOTO / GRAPHIC
 sudo flatpak install -y flathub org.inkscape.Inkscape         ### Vector Graphics Editor

### System 
 sudo flatpak install -y flathub org.mozilla.firefox           ### web browser
 sudo flatpak install -y flathub org.wireshark.Wireshark       ### Network traffic analyzer

# flatpack management

# Search for new Apps
Search Flathub https://flathub.org

# List installed flatpaks
 flatpak list

# List all available applications
 flatpak remote-ls flathub --app

# Update flatpaks
 sudo flatpak update

# Just update one
 sudo flatpak update org.freecad.FreeCAD


###################################
STEP: Use CUDA to verify the GPU

###################################

# Check that the compiler/toolkit is installed
 nvcc -V
# if not install it
 sudo apt update
 sudo apt install nvidia-jetpack
# update your path in your .bashrc file
 gedit ~/.bashrc # Note ~ is a shortcut for the path to your home directory if you are already in your home
# dir you do not need the ~ add the following to the bottom of the .bashrc file
EXAMPLE:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

# resource the .bashrc to update the paths
 source ~/.bashrc
# grab your cuda level from the nvcc -V command (6.2.1 has CUDA 12.6 so there is no cuda_sample-12-6 so use 12-5)
 nvcc -V
 cd ~/github
 git clone --branch v12.5 https://github.com/NVIDIA/cuda-samples.git # Note: using relese 12.5 since there is no 12.6 release
 cd cuda-samples/1_Utilities/deviceQuery # cd to a sample
 make # compile it
 ./deviceQuery # run it

# The following samples have been tired and will work for the Orin Nano super:

 0_Introduction/
    clock
    simpleTexture3D
    vectorAdd

 1_Utilities/
    deviceQuery
    deviceQueryDrv
    bandwidthTest 
 
 3_CUDA_Features/
    simpleCudaGraphs

 5_Domain_Specific/
    BlackScholes   
    marchingCubes
    nbody
    p2pBandwidthLatencyTest
 
 6_Performance/
    transpose


#
#
###################################
STEP: install llama

###################################
#
# Install depends
 sudo apt update && sudo apt install -y git build-essential cmake libcurl4-openssl-dev
# change to local github dir
 cd ~/github
# git clone and compile
 git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
 cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="87"
 cmake --build build --config Release --parallel 8

# Download Command: Use the Hugging Face CLI to download a specific file 
 pip install -U "huggingface_hub[cli]"
# add the path to the huggingface-cli to your .bashrc
 gedit ~/.bashrc
# add the folloing to the end of your .bashrc file
EXAMPLE:

export PATH=~$PATH:~/.local/bin

 source .bashrc

#create a token on hugging face
 https://huggingface.co/settings/tokens
# enter the token
 hf auth login
# download a LLM
 hf download Zhengyi/LLaMA-Mesh




# run
 ./build/bin/llama-server -m path/to/llama-3.2.gguf --n-gpu-layers 40

EXAMPLE:

Recommended Models for Orin Nano Super
(8GB)
Model 	Size	Performance Note
Llama 3.2 1B/3B
	Small/Medium	Best for real-time responsiveness.
Qwen 2.5 3B/7B
	Med/Large	7B requires high quantization (Q4_K_M) and swap.
Gemma 2 2B
	Small	Very efficient for edge use cases.
###EXAMPLE_PATH:


###################################
### Speech to Text to speech
###################################

EXAMPLE:

Setting up an LLM dialogue system with speech on a Jetson Orin Nano Super involves combining three key components: Speech-to-Text (STT) to hear you, a Local LLM to think, and Text-to-Speech (TTS) to talk back. Given the Orin Nano's 8GB memory, the most efficient approach is using containerized, on-device models. 


Prerequisites

    Jetson Orin Nano Super (8GB) with JetPack 5.1.2 or later.
    USB Microphone and Speaker (or a USB headset).
    Swap Space: Set up at least 8GB of swap to handle loading models. 

Step 1: Install Key Technologies
You will need to install AI tools for handling voice and language. The recommended stack for local, low-latency interaction on Orin Nano is Whisper (STT), llama.cpp (LLM), and Piper (TTS). 

    Clone the Assistant Repository:
    bash

    git clone --branch modified https://github.com/ZGMFX20AR/orin_nano_assistant.git
    cd orin_nano_assistant
    chmod +x install_dependencies.sh
    ./install_dependencies.sh

    Set up the Python Environment:
    bash

    python3 -m venv assistant-env
    source assistant-env/bin/activate
    pip install -r requirements.txt

    Build Piper (TTS) and llama.cpp (LLM):
    Follow the specific build instructions provided in the Cytron tutorial for compiling these engines specifically for the ARM64 architecture of the Orin Nano. 



Step 2: Configure Speech & Language Models

    Speech-to-Text (STT): Use OpenAI's Whisper (specifically the tiny or base model for speed).
    LLM Selection: Use optimized small models like Llama-3.2-3B, Gemma-2-2B or Phi-3-mini.
    Text-to-Speech (TTS): Use Piper TTS, which is fast and lightweight for edge devices. 

Configure the assistant.py script to connect these components, updating file paths to your models. 
Step 3: Run the Voice Chat Agent 
Once installed and configured, you can launch the voice-controlled chatbot.

    Activate your virtual environment: source assistant-env/bin/activate
    Run the application: python3 assistant.py

The script will start by initializing the models, then listen for microphone input to send to the LLM and output audio via your speakers. 


Alternative: Using NVIDIA Riva
For a more advanced, production-oriented approach, you can install the NVIDIA Riva SDK. Riva provides high-performance ASR (Automatic Speech Recognition) and TTS pipelines tailored for Jetson, although installation can be complex and consumes significant memory. 
Tips for Success

    Microphone Input: Use arecord -l to identify your microphone and update the DEVICE_NAME (e.g., "plughw:2,0") in your configuration script.
    Performance: If using MLC-LLM rather than llama.cpp, you may achieve better tokens per second (TPS) on Llama-3-8B models.
    Memory Management: The 8GB limit is tight; close unused applications (like the desktop GUI) to free up memory for the model. 








###############################################################################
########                    End-Of-File                                ########
###############################################################################