Category: Blog posts

Update Python easy update to Python 3.9 with homebrew – To update Mac os python from an older version to the latest python example python 3.9.1 you can do the folowing :

This article briefly describes how to replace its version of python on Mac. I wrote a similar article some time ago.

Introduction – update to Python 3.9.1 – HOMEBREW

I used Homebrew to get the latest python as describe on https://formulae.brew.sh/formula/python@3.9:

brew install python@3.9

python --version
Python 3.8.5
python3 --version
Python 3.8.5

You see that it stays on the older version.

Detect where python or python3 is setup

I had to first verify where my python command is pointing, by launching :

which python

It gave me in my case:

/Users/myuser/.pyenv/shims/python

You need also to verify where is python3 command if you have different versions :

which python3  

I get :

/Users/myuser/.pyenv/shims/python3

Replace the version

I need to replace the version of the both command python and python3 by launching :

ln -s -f /usr/local/bin/python3.9 /Users/myuser/.pyenv/shims/python3

And also launch :

ln -s -f /usr/local/bin/python3.9 /Users/myuser/.pyenv/shims/python

The you can test it works :

python --version 
Python 3.9.1
python3 --version 
Python 3.9.1

Mac os update Python – Internal links :

This article briefly describes how to replace its version of python on Mac. I wrote a similar article some time ago.

exiting due to drv_not_detected: no possible driver

😄 minikube v1.17.1 sur Darwin 10.15.7
👎 Unable to pick a default driver. Here is what was considered, in preference order:
▪ parallels : Not installed: exec: “prlctl”: executable file not found in $PATH
▪ podman : Not installed: exec: “podman”: executable file not found in $PATH
▪ virtualbox : Not installed: unable to find VBoxManage in $PATH
▪ vmware : Not installed: exec: “docker-machine-driver-vmware”: executable file not found in $PATH
▪ vmwarefusion : Not installed: the ‘vmwarefusion’ driver is no longer available
▪ docker : Not installed: exec: “docker”: executable file not found in $PATH
▪ hyperkit : Not installed: exec: “hyperkit”: executable file not found in $PATH

❌ Exiting due to DRV_NOT_DETECTED: No possible driver was detected. Try specifying –driver, or see https://minikube.sigs.k8s.io/docs/start/

When you try to launch :

minikube start

You get this error :

😄  minikube v1.17.1 sur Darwin 10.15.7
👎  Unable to pick a default driver. Here is what was considered, in preference order:
    ▪ parallels : Not installed: exec: "prlctl": executable file not found in $PATH
    ▪ podman : Not installed: exec: "podman": executable file not found in $PATH
    ▪ virtualbox : Not installed: unable to find VBoxManage in $PATH
    ▪ vmware : Not installed: exec: "docker-machine-driver-vmware": executable file not found in $PATH
    ▪ vmwarefusion : Not installed: the 'vmwarefusion' driver is no longer available
    ▪ docker : Not installed: exec: "docker": executable file not found in $PATH
    ▪ hyperkit : Not installed: exec: "hyperkit": executable file not found in $PATH

❌  Exiting due to DRV_NOT_DETECTED: No possible driver was detected. Try specifying --driver, or see https://minikube.sigs.k8s.io/docs/start/

Solution setup Docker with homebrew- Unable to pick a default driver :

You can setup Docker with Homebrew Cask : Homebrew Cask is a Homebrew extension for installing GUI software on Mac os like docker.

brew cask install docker

if you get :

Error: Calling `brew cask install` is disabled! Use brew install [--cask] instead.

Then use instead this command and it will works :

brew install docker --cask

Check docker version :

docker --version

If the command doesn’t works :

docker --version
zsh: command not found: docker

So you can run docker app in finder or in command line it will start docker desktop :

open /Applications/Docker.app

Then you can try again minikube :

minikube start

External links – exiting due to drv_not_detected: no possible driver:

https://github.com/Homebrew/homebrew-cask

https://kubernetes.io/fr/docs/setup/learning-environment/minikube/

Internal links :

runtimeerror: attempting to deserialize object on a cuda device but torch.cuda.is_available() is false. if you are running on a cpu-only machine, please use torch.load with map_location=torch.device(‘cpu’) to map your storages to the cpu, How to solve the error presented here with Pytorch in python in the context of deeplearning?

Introduction

First of all, usually this error is encountered when trying to run your model code on the machine’s CPU instead of the GPU.

lib/python3.6/site-packages/torch/serialization.py", ... in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but 
  torch.cuda.is_available() is False. If you are running on a CPU-only machine, 
  please use torch.load with map_location='cpu' to map your storages to the CPU.

In this case you encounter an error which is a raised RuntimeError exception.

This means that you are looking to deserialize an object using code that runs with the GPU while the GPU is disabled.

Explanation – runtimeerror: attempting to deserialize object on a cuda device

Then, The error then occurs when you try to load the pytorch model using the torch.load instruction (see https://pytorch.org/docs/stable/generated/torch.load.html)

Example :

torch.load('mymodel.pt')

So when the reloading is done with the wrong configuration you get the error:

lib/python3.6/site-packages/torch/serialization.py", ... in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but 
  torch.cuda.is_available() is False. If you are running on a CPU-only machine, 
  please use torch.load with map_location='cpu' to map your storages to the CPU.

Solution

Also, As you can see in the Pytorch documentation (see https://pytorch.org/docs/stable/generated/torch.load.html) there is a map_location parameter – (a function, torch.device, string or a dict specifying how to remap storage locations).

This is where we must force the parameter to specify the location to use.

Note that if your model is registered and saved as using the GPU you will have to specify GPU otherwise you will have to put CPU. The objective here is to put a consistent parameter between what has been saved and what is reloaded.

And If you want to force CPU usage :

torch.load('mymodel.pt',map_location='cpu')

For CPU usage you can also use :

torch.load('mymodel.pt', map_location=torch.device('cpu'))

# Load on first GPU
torch.load('mymodel.pt', map_location=lambda storage, loc: storage.cuda(1))
# Lord on GPU 0 and 1
torch.load('mymodel.pt', map_location={'cuda:1':'cuda:0'})

External links :

https://pytorch.org/tutorials/beginner/saving_loading_models.html

https://pytorch.org/tutorials/recipes/recipes/save_load_across_devices.html

https://discuss.pytorch.org/t/saving-and-loading-a-model-in-pytorch/2610

Internal links – runtimeerror: attempting to deserialize object on a cuda device but torch.cuda.is_available() is false :

https://128mots.com/index.php/en/category/non-classe-en/

There are several approaches to recording (serializing) and loading (deserialize) patterns for inference in PyTorch.

For example you may need to load a model that is already trained and back up that comes from the internet. More recently I answered this question on a discussion forum https://discuss.pytorch.org/t/i-want-to-do-machine-learning-with-android/98753. I take advantage of this article to give some details.

SAVE AND LOAD OF A PRE-TRAINED MODE with load_state_dict

In PyTorch you can save a model by storing in its file its state_dict these are Python dictionaries, they can be easily recorded, updated, modified and restored, adding great modularity to PyTorch models and optimizers.

In my example:

ESPNet is backed up with this method

https://github.com/sacmehta/ESPNet/tree/master/pretrained/encoder

I managed to load the encoder model using the ESPNet class that makes a load_state_dict

import torch
model = ESPNet(20,encoderFile="espnet_p_2_q_8.pth", p=2, q=8)
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt")

The exit is

Encode loaded!

Note that ESPNet uses the default GPU and that an adaptation in Model.py in https://github.com/sacmehta/ESPNet/tree/master/train was required by replacing:

self.encoder.load_state_dict(torch.load(encoderFile)

By:

self.encoder.load_state_dict(torch.load(encoderFile,map_location='cpu'))

Otherwise if you don’t make this adjustment don’t have the ability to launch on a GPU you’ll get the following error:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location-torch.device ('cpu') to map your storages to the CPU.

BACKUP AND LOAD BY PYTORCH SCRIPT

https://pytorch.org/docs/stable/jit.html

TorchScript is a way to create models that can be made that can be optimized from the PyTorch code.

Any TorchScript program can be saved from a Python process and loaded into a process where there is no Python dependency.

The code below allows you to save the pre-trained ESPNet model that was backed up by the classic torch.save method via the use of TorchScript.

import torch


model = ESPNet(20,encoderFile="espnet_p_2_q_8.pth", p=2, q=8)
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
#Exemple de save avec TorchScript
torch.jit.save(traced_script_module, "scriptmodel.pth")

An example for loader via TorchScript below:

import torch
model = torch.jit.load("scriptmodel.pth")

RESOURCES ON THE SUBJECT

https://pytorch.org/docs/stable/jit.html

https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html

https://stackoverflow.com/questions/53900396/what-are-torch-scripts-in-pytorch

The problem of the algorithmic backpack is interesting and is part of the first digital science and computer science program.

This problem illustrates the gluttonous algorithms that list all the possibilities of solving a problem to find the best solution.

The problem of the backpack is a problem of optimization, i.e. a function that must be maximized or minimized and constraints that must be met.

The problem of the backpack

For a backpack of maximum capacity of P and N items each with its own weight and a set value, throw the items inside the backpack so that the final contents have maximum value.

Example of a statement:

• Maximum backpack capacity: 11 units
• Item number: 5
• Object Values: $10.50,20,30.60
• Weight of Objects: '1,5,3,2,4'

What is the maximum value that can be put in the backpack considering the maximum capacity constraint of the bag which is 11?

Gluttonous algorithm

An effective solution is to use a gluttonous algorithm. The idea is to calculate the value/weight ratio for each object and sort the object based on this calculated ratio.

You take the object with the highest ratio and add until you can't add any more.

In fractional version it is possible to add fractions of item to the backpack.

Implementation of the problem in non-fractional version

Here is an implementation of the problem in a non-fractional version, i.e. you can't add a fraction of an object to the bag. Only whole objects can be added.

itemsac class: 
    def __init__ (self, weight, value, index): 
        self.index - index         
        self.weight - weight 
        self.value
        self.report - value // weight 
  #Fonction for comparison between two ObjectsSac
  #On compares the ratio calculated to sort them
    def __lt__ (self, other): 
        return self.report< other.rapport 
  

def getValeurMax (weights, values, ability): 
        TableTrie[] 
        for i in range: 
            tableTrie.append (ObjectSac (weigh[i]ts, value[i]s, i)) 
  
        #Trier the elements of the bag by their report
        tableTrie.sort (reverse - True) 
  
        MeterValeur - 0
        for object in tableauTrie: 
            WeightCourant - int (object.weight) 
            ValueCourante - int (object.value) 
            if capacity - weightsCourant '0: 
                #on adds the object to the bag
                #On subtracts capacity
                capacity - weightsCourant 
                MeterValeur - ValueCourante
                #On adds value to the bag 
        return counterValeur 


Weights[1,5,3,2,4] 
Values[10,50,20,30,60] 
capacity - 11
Maximum value - getValeurMax (weight, values, capacity) 
print ("Maximum value in the backpack," ValueMax) 

The result is as follows:

py sacados.py 
Maximum value in backpack - 120

Implementation of the problem in fractional version

In a fractional version of the backpack problem you can add fractions of object to the backpack.

itemsac class: 
    def __init__ (self, weight, value, index): 
        self.index - index         
        self.weight - weight 
        self.value
        self.report - value // weight 
  #Fonction for comparison between two ObjectsSac
  #On compares the ratio calculated to sort them
    def __lt__ (self, other): 
        return self.report< other.rapport 
  

def getValeurMax (weights, values, ability): 
        TableTrie[] 
        for i in range: 
            tableTrie.append (ObjectSac (weigh[i]ts, value[i]s, i)) 
  
        #Trier the elements of the bag by their report
        tableTrie.sort (reverse - True) 
  
        MeterValeur - 0
        for object in tableauTrie: 
            WeightCourant - int (object.weight) 
            ValueCourante - int (object.value) 
            if capacity - weightsCourant '0: 
                #on adds the object to the bag
                #On subtracts capacity
                capacity - weightsCourant 
                MeterValeur - ValueCourante
                #On adds value to the bag 
            else 
                fraction - capacity / weightCouring 
                MeterValeur -Courante value - fraction 
                capacity - int (capacity - (weightsCourant - fraction)) 
                Break
        return counterValeur 


Weights[1,5,3,2,4] 
Values[10,50,20,30,60] 
capacity - 11
Maximum value - getValeurMax (weight, values, capacity) 
print ("Maximum value in the backpack," ValueMax) 

The result is as follows:

py sacados.py 
Maximum value in backpack - 140.0