ONNX export issues

[Demilock]

This is a great model with repetitive refining of the mask. But please implement the code for ONNX exporting. I am trying to export but it is very difficult to figure out the exact inputs that are going to the model. Through inspection i found that for 1 image and 1 click, the model's forward method takes 2 batches [2,3,H,W] and then 2 clicks [2,N,3]. Also why are clicks appended with [-1,-1,-1] before sending to forward()? Also, since ISModel is inherited by PlainVitModel and real forward for the end to end input and output is backbone_forward() of PlainVitModel and not forward() of ISModel, it is very difficult to get the correct forward methods to get to work. I have to manually preprocess and postprocess the inputs and outputs and for that i have to know what exactly is the input shapes, types and meaning.
Even if you explain it here, it would be very helpful and maybe i can include ONNX export code to the repository saving you time and work. Or maybe you can include the ONNX export code here yourself.

[qinliuliuqin]

@Demilock Thanks for bring it to my attention. This is also related to #37. I will get you back by the end of this week.

[qinliuliuqin]

@Demilock Hi, could you please include your ONNX export code here? I realized that it's actually more complex than I expected and your code might save my time. Thank you in advance.

[Demilock]

run_ONNX.py

export_ONNX.py

[Demilock]

Above comment has code files to both export and then run the exported ONNX model. You can paste them at the same location where demo.py is located. some notes about the files:

-export_ONNX.py
you can adjust the backbone, head, neck parameters. you can turn "withPrevMask" on/off. You have to adjust the "ckpt_path" to where you have the cocolvis_vit_base.pth file. Since i was not able to run the forward method of ISModel, i had to wrap it inside a Wrapper class to explicitly call backbone_forward method. Hence, now the ONNX model takes input the input image and the coord_features of the clicks and not image and plain clicks. The model will be exported with dynamic parameters enabled meaning we can feed any number of batches and height and width.

-run_ONNX.py
this will run your exported ONNX model with a preprocessed image and dummy clicks. It will save the mask and the overlay image. Since i dont know how clicks should be preprocessed, currently the output is rubbish. You have to adjust the clicks preprocessing to run the model correctly. Currently the click preprocessing is just copied from the repo code but not accurately enough. Currently this file is just a poor attempt and needs modifications.

Thank you.

[Demilock]

Hello, did you get a chance to look into this? i've tried everything to preprocess like the repo but to no avail. in the end i even saved and loaded the preprocessed tensors from repo to my run_ONNX.py but that also gives out weird looking masks. Please check this issue if you get the time.

Thank you.

[qinliuliuqin]

@Demilock Thank you for your hard work and dedication. I’m not sure if I’ll have enough time to complete this, but I’ll keep you updated on any progress I make.