PyTorch how to load a pre-trained model?

There are several approaches to recording (serializing) and loading (deserialize) patterns for inference in PyTorch.

For example you may need to load a model that is already trained and back up that comes from the internet. More recently I answered this question on a discussion forum https://discuss.pytorch.org/t/i-want-to-do-machine-learning-with-android/98753. I take advantage of this article to give some details.

SAVE AND LOAD OF A PRE-TRAINED MODE with load_state_dict

In PyTorch you can save a model by storing in its file its state_dict these are Python dictionaries, they can be easily recorded, updated, modified and restored, adding great modularity to PyTorch models and optimizers.

In my example:

ESPNet is backed up with this method

https://github.com/sacmehta/ESPNet/tree/master/pretrained/encoder

I managed to load the encoder model using the ESPNet class that makes a load_state_dict

import torch
model = ESPNet(20,encoderFile="espnet_p_2_q_8.pth", p=2, q=8)
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt")

The exit is

Encode loaded!

Note that ESPNet uses the default GPU and that an adaptation in Model.py in https://github.com/sacmehta/ESPNet/tree/master/train was required by replacing:

self.encoder.load_state_dict(torch.load(encoderFile)

By:

self.encoder.load_state_dict(torch.load(encoderFile,map_location='cpu'))

Otherwise if you don’t make this adjustment don’t have the ability to launch on a GPU you’ll get the following error:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location-torch.device ('cpu') to map your storages to the CPU.

BACKUP AND LOAD BY PYTORCH SCRIPT

https://pytorch.org/docs/stable/jit.html

TorchScript is a way to create models that can be made that can be optimized from the PyTorch code.

Any TorchScript program can be saved from a Python process and loaded into a process where there is no Python dependency.

The code below allows you to save the pre-trained ESPNet model that was backed up by the classic torch.save method via the use of TorchScript.

import torch


model = ESPNet(20,encoderFile="espnet_p_2_q_8.pth", p=2, q=8)
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
#Exemple de save avec TorchScript
torch.jit.save(traced_script_module, "scriptmodel.pth")

An example for loader via TorchScript below:

import torch
model = torch.jit.load("scriptmodel.pth")

RESOURCES ON THE SUBJECT

https://pytorch.org/docs/stable/jit.html

https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html

https://stackoverflow.com/questions/53900396/what-are-torch-scripts-in-pytorch

BCEWithLogitsLoss Pytorch with python

BCEWithLogitsLoss Here are some additional explanations on using Binary Cross-Entropy Loss with Pytorch in python.

Introduction

I have been asked recently on How to find the code BCEWithLogitsLoss (https://discuss.pytorch.org/t/implementation-of-binary-cross-entropy/98715/2) for a function called in PYTORCH with the function handle_torch_function it takes me some time to understand it, and i will share it with you if it helps:

The question was about BCEWithLogitsLoss = BCELoss + sigmoid() ? My answer can be apply if you want to analyse the code of all the functions that figures in the ret dictionnary from https://github.com/pytorch/pytorch/blob/master/torch/overrides.py

BCEWithLogitsLoss is a combination of BCELOSS + a Sigmoid layer i. This is more numerically stable than using a plain Sigmoid followed by a BCELoss as, by combining the operations into one layer, it takes advantage of the log-sum-exp trick for numerical stability see : https://en.wikipedia.org/wiki/LogSumExp

BCEWithLogitsLoss in details

  1. The code of the BCEWithLogitsLoss Class can be found in https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/loss.py
    def forward(self, input: Tensor, target: Tensor) -> Tensor:
        return F.binary_cross_entropy_with_logits(input, target,
                                                  self.weight,
                                                  pos_weight=self.pos_weight,
                                                  reduction=self.reduction)

The F oject is imported from functionnal.py here : https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py

You will find the function called

def binary_cross_entropy_with_logits(input, target, weight=None, size_average=None,
                                     reduce=None, reduction='mean', pos_weight=None):

It calls the handle_torch_function in https://github.com/pytorch/pytorch/blob/master/torch/overrides.py
You will find an entry of the function binary_cross_entropy_with_logits in the ret dictionnary wich contain every function that can be overriden in pytorch.
This is the Python implementation of torch_function
More info in https://github.com/pytorch/pytorch/issues/24015

Then the code called is in the C++ File
https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Loss.cpp

Tensor binary_cross_entropy_with_logits(const Tensor& input, const Tensor& target, const Tensor& weight, const Tensor& pos_weight, int64_t 
...

LIENS DE RESSOURCES :

https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html

https://discuss.pytorch.org/t/bceloss-vs-bcewithlogitsloss/33586

https://stackoverflow.com/questions/59246416/the-output-of-numerical-calculation-of-bcewithlogitsloss-for-pytorch-example

https://www.semicolonworld.com/question/61244/implementing-bcewithlogitsloss-from-pytorch-in-keras

https://128mots.com/index.php/en/2020/04/10/build-an-application-decentralises-full-stack-not-a-not-ethereum-blockchain-dapp-en-plus-de-128-words-part-3-2/

https://128mots.com/index.php/en/category/non-classe-en/

Mastering IBM COBOL Language – Part 1 – FREE TRAINING

This article follows my introduction to the COBOL language. I advise you to reread the first part that deals with the general principles of COBOL. Here: https://128mots.com/index.php/2020/10/07/ibm-cobol/ and English https://128mots.com/index.php/en/2020/10/07/ibm-cobol-3/ (IBM COBOL FREE TRAINING)

Here we discuss the general concepts of the structure of a COBOL program.

DATA DIVISION

All the data that will be used by the program is located in the Data Division. This is where all the memory allowances required by the program are taken care of. This division is optional.

FILE MANAGEMENT:

FILE SECTION describes data sent or from the system, especially files.

The SYNtax in COBOL is the following

DATA DIVISION.
FILE SECTION.
 FD/SD NameOfFile[RECORD CONTAINS intgr CHARACTERS] 
	[BLOCK CONTAINS intgr RECORDS] 
[DATA RECORD IS NameOfRecord].
	[RECORDING MODE IS {F/V/U/S}]

FD describes the files and SD the sorting files.

FILE IN INPUT

In FILE-CONTROL the statement will be:

           SELECT FMASTER ASSIGN FMASTER
                  FILE STATUS W-STATUS-FMASTER.

If the input file is indexed:
           SELECT FMASTER ASSIGN FMASTER
                  ORGANIZATION IS INDEXED
                  RECORD KEY IS FMASTER-KEY
                  FILE STATUS W-STATUS-FMASTER.

In this case at the file-SECTION level we will have:

      FMAITRE as an appetizer  
       FD FMAITRE.
       01 ENR-FMAITRE.
      Statements from the registration areas

At the JCL level the statement will be of the form:
ENTREE DD DSN-SAMPLE. INPUTF,DISP-SHR

FILE OUT

The JCL statement will then be:

OUTFILE DD DSN-SAMPLE. OUTPUTF,DISP(,CATLG,DELETE),
LRECL-150,RECFM-FB

RECFM specifies the characteristics of records with fixed length (F), variable length (V), variable length ASCII (D) or indefinite length (U). Records that are said to be blocked are described as FB, VB or DB.

OPENING AND CLOSING FILE IN PROCEDURE DIVISION

COBOL uses PROCEDURE DIVISION mechanisms to perform writing, closing and file openings.

Entry file opening:

OPEN INPUT FICENT

Opening the output file:

OPEN OUTPUT FICSOR

File closure:

CLOSE FICENT
CLOSE FICSOR

File playback:

READ ACTENR
AT END MOVE 'O' TO LAST-RECORD
END-READ

File writing

WRITE SOR-ENR

EXTERNAL LINKS TO RESOURCES (IBM COBOL FREE TRAINING)

Below are some links that I found interesting that also deal with file management with COBOL.

Example of IBM file management: https://www.ibm.com/support/knowledgecenter/en/SS6SGM_5.1.0/com.ibm.cobol51.aix.doc/PGandLR/ref/rpfio13e.html

REFM Format: https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.2.0/com.ibm.zos.v2r2.idad400/d4037.htm

Tutorialspoint an article on file management https://www.tutorialspoint.com/cobol/cobol_file_handling.htm

COURSERA COBOL with VSCODE: https://www.coursera.org/lecture/cobol-programming-vscode/file-handling-YVlcf

MEDIUM an interesting article for COBOL beginners: https://medium.com/@yvanscher/7-cobol-examples-with-explanations-ae1784b4d576

Also on his blog: http://yvanscher.com/2018-08-01_7-cobol-examples-with-explanations–ae1784b4d576.html

Many example COBOL free tutorials and sample code: http://www.csis.ul.ie/cobol/examples/default.htm

GITHUB Awesome-cobol https://github.com/mickaelandrieu/awesome-cobol you

https://128mots.com/index.php/2020/10/07/ibm-cobol/

IBM COBOL FREE TRAINING

How do I make a career in COBOL?

COBOL (COmmon Business Oriented Language) is a business-oriented language it is very suitable for data processing with high performance and precision, it is offered by IBM.

You can be sure that any purchases or withdrawals you make with your credit card start a COBOL program. Every day COBOL processes millions of transactions. From my point of view learning COBOL is a very interesting set of knowledge.

I have worked with this language for more than 10 years and I share here some notes that will probably allow you to get your hands on it.

The mainframe is modernizing with the ability to program with the latest tools such as the free code editor VsCode and zowe extension as well as Z Open editor, run in the cloud in environments such as Open Shift and integrate devops principles with tools such as Github, Jenkins, Wazi.

COBOL Syntax

The SYNtax of COBOL is quite simple and is similar to the natural language in English.

The code is standardized according to columns that can describe 5 key areas.

Area sequence: specifies a sequence number of the line of code sometimes, sometimes blank

Indicator Area: May contain an indicator for example – to indicate that the line is a comment, D to indicate that the line runs only in debugging mode.

A AREA: Contains divisions, sections, paragraphs and Lift

B AREA: Sentences and statements of the program cobol for example COMPUTE something…

Area Identification: space to ignore and leave blank.

There are also words reserved in COBOL you will find on the link the list of words reserved in COBOL. https://www.ibm.com/support/knowledgecenter/SSZJPZ_9.1.0/com.ibm.swg.im.iis.ds.mfjob.dev.doc/topics/r_dmnjbref_COBOL_Reserved_Words.html

Divisions:

The code is structured by divisions that contain Paragraph-composed Sections themselves made up of Sentences and Statements.

Example of sentences:

ADD 45 TO PRICE.

Note that the point corresponds to an implied scope terminator.

There are 4 Divisions in a COBOL program:

– DATA DIVISION: allows you to set up the data management that will be processed by the program.

– DIVISION IDENTIFICATION: Program name and programmer, program date, program purpose.

– QUARTER ENVIRONEMENT: Type of computer uses and mapping between files used in the program and dataset on the system (link between program and system)

– PROCEDURE DIVISION: This is where the business code composed of the different paragraphs to be executed is contained.

The variables in Cobol:

As in other languages, letters can be used to represent values stored in memory.

The name of a variable is a maximum of 30 characters.

A Picture clause allows you to determine the type of variable.

PIC 9: Digital length is in brackets.

PIC 9(5): digital variable of 5 digits the maximum length for a digital is 18.

PIC A for a character

PIC X(11): an alphanumeric with a maximum length of 255

It is possible to have types edited using symbols:

PIC 9(6)V99 for a 6 digits and 2 decimals separated by comma.

PIC $9,999V99 to represent an amount

Note that COBOL provides constant liters such as ZEROES, SPACE, SPACES, LOW-VALUE …

More information on this link:

https://www.ibm.com/support/knowledgecenter/SS6SG3_4.2.0/com.ibm.entcobol.doc_4.2/PGandLR/ref/rllancon.htm

If you are new to IBM COBOL and want to do a serious and not too expensive apprenticeship, I recommend you read this book:

This book covers a lot of topics related to machine language, IBM Cobol training, Open Cobol IDE, DB2 to become a true Cobol programmers.

See also my articles:

How did I prepare the PRINCE2® Practitioner certification?

I have just obtained the PRINCE2® Practitionner certification. 

PRINCE2® is a project management methodology used around the world that can adapt to all types of projects.

With an average salary of $84,450 for a certified project manager, (here), I have to say that PRINCE2® is from my point of view a very interesting skill set to get.

Learning PRINCE2® is a bit like saying “I want to work with the best method to manage and control projects with an international scope.»

I share here some notes and my feedback as well as resources, which you might be interested in if you are considering certifying PRINCE2®.

The first step in certification is the PRINCE2® Fundamental (PRINCE2® Foundation exam, which validates that you have the knowledge to participate in a project that uses the PRINCE2®The PRINC
E2® Practitioner (PRINCE2® Practitioner) exam aims for a perfect mastery of the method for managing and managing a project, the Fundamental exam is a prerequisite.

I would advise you to read the official book Managing Successful Projects with PRINCE2®

Available here:

and you are entitled to it during the PRINCE2® Practitioner exam.
You can choose to pass the certification in French or English. I advise you to pass it in English because in my opinion it gives a more international character to the certification. You will also have an additional 30 minutes to take the exam if you do not take it in your native language.

Here’s an interesting resource list I found online:

https://prince2.wiki: Free media to help you pass PRINCE2® certifications.

Free online exam simulator for PRINCE2® Foundation https://mplaza.training/exam-simulators/prince2-foundation/

Free online exam simulator for PRINCE2® Practitioner https://mplaza.training/exam-simulators/prince2-practitioner/

The official website PRINCE2®de Axelos: https://www.axelos.com/best-practice-solutions/prince2

If you are looking for a book in French I recommend this one:

Intellectual Property :
PRINCE2® and PRINCE2 Agile® are registered trade marks of AXELOS Limited, used under permission of AXELOS Limited. All rights reserved.