It would be nice to support Metal Performance Shaders backend for pytorch on macOS the main changes for inference seem to be: - `load_in_8bit` must be `False` - `device_map` set to `{"":"mps"}` see: https://github.com/Birch-san/falcon-play/blob/cbf9b2aebe7eef3eea305a511d6cdda17282ca8a/scripts/chat_play.py#L154-L157 https://github.com/Birch-san/falcon-play/blob/cbf9b2aebe7eef3eea305a511d6cdda17282ca8a/scripts/chat_play.py#L175 cc @Birch-san are there any other good docs or resources for adding mps support?
It would be nice to support Metal Performance Shaders backend for pytorch on macOS
the main changes for inference seem to be:
load_in_8bitmust beFalsedevice_mapset to{"":"mps"}see:
https://github.com/Birch-san/falcon-play/blob/cbf9b2aebe7eef3eea305a511d6cdda17282ca8a/scripts/chat_play.py#L154-L157
https://github.com/Birch-san/falcon-play/blob/cbf9b2aebe7eef3eea305a511d6cdda17282ca8a/scripts/chat_play.py#L175
cc @Birch-san are there any other good docs or resources for adding mps support?