make apple-isp processed V4L2 capture node usable through libcamera

[FreddyFunk]

The apple-isp driver already exposes processed NV12 frames through a standard, single-node V4L2 capture interface. For libcamera support a pipeline handler need to explicit claim it.

The existing libcamera uvcvideo pipeline handler appears suitable for this type of firmware-processed camera. The apple-isp driver likely only needs to expose the media-entity metadata expected by that handler. That means:

• register /dev/video0 as an entity belonging to the apple-isp media device
• mark that entity with MEDIA_ENT_FL_DEFAULT so libcamera knows it is the capture node to open
• give the entity an appropriate capture sink pad
• report consistent media-device and VIDIOC_QUERYCAP information
  • driver: apple-isp
  • model/card: FaceTime HD Camera
  • stable platform bus information
• report a coherent single-planar capture API
  • V4L2_CAP_VIDEO_CAPTURE
  • V4L2_CAP_STREAMING
  • NV12 formats, sizes, stride and buffer sizes that agree with the queue implementation

Once that lands, libcamera could whitelist apple-isp and use the camera identification instead of USB specific metadata.

This should allow libcamera applications to use the existing processed output without implementing a complete apple specific pipeline handler. A full apple-isp handler in libcamera could be a next step once this driver is upstreamed to mainline Linux.

I currently do not have access to Apple M-series hardware. Otherwise I would be able to develop, test and submit a proper patch for this myself. I'm reaching out here since I had a user that reported a non working webcam on my libcamera only app https://github.com/cosmic-utils/cosmic-ext-camera

[sofus13]

If this is the report cosmic-utils/cosmic-ext-camera#643, i think https://github.com/patjak/facetimehd is the place you are looking for

[FreddyFunk]

No, it's related to this report:
https://gitlab.postmarketos.org/postmarketOS/pmaports/-/merge_requests/9531#note_584080

[sofus13]

please note that i had to make some changes to the uvcvideo pipeline.

diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
index 4b09bd6e2..6c54d74c3 100644
--- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
+++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
@@ -470,7 +470,7 @@ int PipelineHandlerUVC::queueRequestDevice(Camera *camera, Request *request)
 bool PipelineHandlerUVC::match(DeviceEnumerator *enumerator)
 {
        std::shared_ptr<MediaDevice> media;
-       DeviceMatch dm("uvcvideo");
+       DeviceMatch dm("apple-isp");
 
        media = acquireMediaDevice(enumerator, dm);
        if (!media)
@@ -646,6 +646,8 @@ bool UVCCameraData::generateId()
                if (!controllerId.empty())
                        break;
        }
+         id_ = controllerId;
+         return true;
 
        /*
         * Create a USB ID from the device path which has the known format:

[FreddyFunk]

-       DeviceMatch dm("uvcvideo");
+       DeviceMatch dm("apple-isp");

This libcamera patch will break a lot of USB webcams.

Can you try the following libcamera patch together with your kernel patch from #619 and test if with both the internal webcam and USB webcams connected to the Apple device?

diff --git a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
index 3435a7604..4c3099660 100644
--- a/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
+++ b/src/libcamera/pipeline/uvcvideo/uvcvideo.cpp
@@ -62,7 +62,7 @@ public:
 	std::optional<v4l2_exposure_auto_type> manualExposureMode_;
 
 private:
-	bool generateId();
+	bool generateId(const std::string &driver);
 
 	std::string id_;
 };
@@ -470,9 +470,14 @@ int PipelineHandlerUVC::queueRequestDevice(Camera *camera, Request *request)
 bool PipelineHandlerUVC::match(DeviceEnumerator *enumerator)
 {
 	std::shared_ptr<MediaDevice> media;
-	DeviceMatch dm("uvcvideo");
 
-	media = acquireMediaDevice(enumerator, dm);
+	for (const char *driver : { "uvcvideo", "apple-isp" }) {
+		DeviceMatch dm(driver);
+		media = acquireMediaDevice(enumerator, dm);
+		if (media)
+			break;
+	}
+
 	if (!media)
 		return false;
 
@@ -535,7 +540,7 @@ int UVCCameraData::init(std::shared_ptr<MediaDevice> media)
 	video_->bufferReady.connect(this, &UVCCameraData::imageBufferReady);
 
 	/* Generate the camera ID. */
-	if (!generateId()) {
+	if (!generateId(media->driver())) {
 		LOG(UVC, Error) << "Failed to generate camera ID";
 		return -EINVAL;
 	}
@@ -584,7 +589,9 @@ int UVCCameraData::init(std::shared_ptr<MediaDevice> media)
 	 * come from the ACPI _PLD, but that may be even more unreliable than
 	 * the _UPC.
 	 */
-	properties::LocationEnum location = properties::CameraLocationExternal;
+	properties::LocationEnum location = media->driver() == "apple-isp"
+						    ? properties::CameraLocationFront
+						    : properties::CameraLocationExternal;
 	std::ifstream file(video_->devicePath() + "/../removable");
 	if (file.is_open()) {
 		std::string value;
@@ -626,7 +633,7 @@ int UVCCameraData::init(std::shared_ptr<MediaDevice> media)
 	return 0;
 }
 
-bool UVCCameraData::generateId()
+bool UVCCameraData::generateId(const std::string &driver)
 {
 	const std::string path = video_->devicePath();
 
@@ -647,6 +654,12 @@ bool UVCCameraData::generateId()
 			break;
 	}
 
+	/* Platform devices don't have USB topology or vendor/product IDs. */
+	if (driver == "apple-isp") {
+		id_ = controllerId;
+		return true;
+	}
+
 	/*
 	 * Create a USB ID from the device path which has the known format:
 	 *
-- 

[sofus13]

using cam -l it shows up as /base/soc. I dont have an USB webcam