This article demonstrates the detection of different facial gestures (Head Nods, Eye Blinks, Smile etc) with the help of Firebase ML Kit Face Detection API. Here we will be mainly focusing on the use of Firebase ML Kit Vision API to detect different facial gestures. For initial setup of the project you can visit the Part One of this series.
Face Detection Using ML KitWith ML Kits face detection API, you can detect faces in an image, identify key facial features, and get the contours of detected faces.With face detection, you can get the information you need to perform tasks like embellishing selfies and portraits, or generating avatars from a users photo. Because ML Kit can perform face detection in real time, you can use it in applications like video chat or games that respond to the players expressions.
You can know more about Firebase Face Detection API by clicking the link here.Lets not waste anymore time and get started.TutorialIn the Part One of the series we have completed the initial setup of the application.
In this article we will see how to make use of Firebase MLVision API to detect different facial gestures.Creating a FacialGestureCameraView.swift FileLets create a FacialGestureCameraView.
swift file which is a subclass of UIView class and lets import the below frameworks in the header of the file. import AVFoundationimport FirebaseMLVision2. Then lets create the below threshold variables to determine different facial gestures.
public var leftNodThreshold: CGFloat = 20. 0public var rightNodThreshold: CGFloat = -4public var smileProbality: CGFloat = 0.8public var openEyeMaxProbability: CGFloat = 0.
95public var openEyeMinProbability: CGFloat = 0. 1private var restingFace: Bool = trueThere is no need to explain about this variables as it is self explanatory.3.
Lets create few more lazy variables which will be used for computing facial gestures as shown below.private lazy var vision: Vision = return Vision. vision()()private lazy var options: VisionFaceDetectorOptions = let option = VisionFaceDetectorOptions()option.
performanceMode = .accurateoption.landmarkMode = .
noneoption.classificationMode = .alloption.
isTrackingEnabled = falseoption.contourMode = .nonereturn option()private lazy var videoDataOutput: AVCaptureVideoDataOutput = let videoOutput = AVCaptureVideoDataOutput()videoOutput.
alwaysDiscardsLateVideoFrames = truevideoOutput. setSampleBufferDelegate(self, queue: videoDataOutputQueue)videoOutput.connection(with: .
video)?.isEnabled = truereturn videoOutput()private let videoDataOutputQueue: DispatchQueue = DispatchQueue(label: Constants.
videoDataOutputQueue)private lazy var previewLayer: AVCaptureVideoPreviewLayer = let layer = AVCaptureVideoPreviewLayer(session: session)layer. videoGravity = . resizeAspectFillreturn layer()private let captureDevice: AVCaptureDevice?
= AVCaptureDevice.default(.builtInWideAngleCamera, for: .
video, position: .front)private lazy var session: AVCaptureSession = return AVCaptureSession()()4. Now lets write logic to begin and end the session as shown below.
func beginSession() guard let captureDevice = captureDevice else return guard let deviceInput = try? AVCaptureDeviceInput(device: captureDevice) else return if session.canAddInput(deviceInput) session.
addInput(deviceInput)if session.canAddOutput(videoDataOutput) session.addOutput(videoDataOutput)layer.
masksToBounds = truelayer.addSublayer(previewLayer)previewLayer.frame = boundssession.
startRunning()func stopSession() session.stopRunning() 5. Now lets implement AVCaptureVideoDataOutputSampleBufferDelegate delegate method and its dependent methods as shown below.
public func captureOutput(_ output: AVCaptureOutput,didOutput sampleBuffer: CMSampleBuffer,from connection: AVCaptureConnection) guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else print("Failed to get image buffer from sample buffer.")returnlet visionImage = VisionImage(buffer: sampleBuffer)let metadata = VisionImageMetadata()let visionOrientation = visionImageOrientation(from: imageOrientation())metadata.orientation = visionOrientationvisionImage.
metadata = metadatalet imageWidth = CGFloat(CVPixelBufferGetWidth(imageBuffer))let imageHeight = CGFloat(CVPixelBufferGetHeight(imageBuffer))DispatchQueue.global().async self.
detectFacesOnDevice(in: visionImage,width: imageWidth,height: imageHeight)private func visionImageOrientation(from imageOrientation: UIImage.Orientation) ->VisionDetectorImageOrientation switch imageOrientation case . up:return .
topLeftcase . down:return . bottomRightcase .
left:return . leftBottomcase . right:return .
rightTopcase . upMirrored:return . topRightcase .
downMirrored:return . bottomLeftcase . leftMirrored:return .
leftTopcase . rightMirrored:return . rightBottom@unknown default:fatalError()private func imageOrientation(fromDevicePosition devicePosition: AVCaptureDevice.
Position = .front) -> UIImage.Orientation var deviceOrientation = UIDevice.
current.orientationif deviceOrientation == .faceDown ||deviceOrientation == .
faceUp ||deviceOrientation == .unknown deviceOrientation = currentUIOrientation()switch deviceOrientation case . portrait:return devicePosition == .
front ? . leftMirrored : .
rightcase . landscapeLeft:return devicePosition == . front ?
. downMirrored : . upcase .
portraitUpsideDown:return devicePosition == .front ? .
rightMirrored : .leftcase .landscapeRight:return devicePosition == .
front ? .upMirrored : .
downcase .faceDown, .faceUp, .
unknown:return .up@unknown default:fatalError()private func currentUIOrientation() -> UIDeviceOrientation let deviceOrientation = () -> UIDeviceOrientation inswitch UIApplication.shared.
windows.first?.
windowScene?.interfaceOrientation case .
landscapeLeft:return .landscapeRightcase .landscapeRight:return .
landscapeLeftcase .portraitUpsideDown:return .portraitUpsideDowncase .
portrait, .unknown, .none:return .
portrait@unknown default:fatalError()guard Thread. isMainThread else var currentOrientation: UIDeviceOrientation = .portraitDispatchQueue.
main.sync currentOrientation = deviceOrientation()return currentOrientationreturn deviceOrientation()6. Now lets create delegates which will be triggered when a particular gesture will be detected as shown below.
@objc public protocol FacialGestureCameraViewDelegate: class @objc optional func doubleEyeBlinkDetected()@objc optional func smileDetected()@objc optional func nodLeftDetected()@objc optional func nodRightDetected()@objc optional func leftEyeBlinkDetected()@objc optional func rightEyeBlinkDetected()7. Now lets create a delegate object in the FacialGestureCameraView class which needs to be confirmed to implement the delegate methods as shown below.public weak var delegate: FacialGestureCameraViewDelegate?
8. Now lets write the most important method where the face gesture detection logic has been implemented.private func detectFacesOnDevice(in image: VisionImage, width: CGFloat, height: CGFloat) let faceDetector = vision.
faceDetector(options: options)faceDetector. process(image, completion: features, error inif let error = error print(error.localizedDescription)returnguard error == nil, let features = features, !
features.isEmpty else returnif let face = features. first let leftEyeOpenProbability = face.
leftEyeOpenProbabilitylet rightEyeOpenProbability = face.rightEyeOpenProbability// left head nodif face.headEulerAngleZ > self.
leftNodThreshold if self.restingFace self.restingFace = falseself.
delegate?.nodLeftDetected?
() else if face.headEulerAngleZ
restingFace self. restingFace = falseself. delegate?
. nodRightDetected?() else if leftEyeOpenProbability > self.
openEyeMaxProbability &&rightEyeOpenProbability
restingFace = falseself. delegate?.
rightEyeBlinkDetected?() else if rightEyeOpenProbability > self.openEyeMaxProbability &&leftEyeOpenProbability
openEyeMinProbability // Left Eye Blinkif self. restingFace self. restingFace = falseself.
delegate?. leftEyeBlinkDetected?
() else if face. smilingProbability > self. smileProbality // smile detectedif self.
restingFace self. restingFace = falseself. delegate?
. smileDetected?() else if leftEyeOpenProbability
openEyeMinProbability && rightEyeOpenProbability
restingFace = falseself. delegate?.
doubleEyeBlinkDetected?() else // Face got resetedself.restingFace = true)I know the article is getting lengthy, but we are almost done with our logic.
The only thing which is pending is to implement the delegate methods from our ViewController. swift class. Lets implement that as well.
Implementing Logic In ViewController. swift FileIn this file we need to implement FacialGestureCameraViewDelegate methods so that we will receive callbacks when a particular facial gesture is detected. Create an extension of ViewController and implement the delegate methods as shown below.
extension ViewController: FacialGestureCameraViewDelegate func doubleEyeBlinkDetected() print("Double Eye Blink Detected")func smileDetected() print("Smile Detected")func nodLeftDetected() print("Nod Left Detected")func nodRightDetected() print("Nod Right Detected")func leftEyeBlinkDetected() print("Left Eye Blink Detected")func rightEyeBlinkDetected() print("Right Eye Blink Detected")2. Add the remaining code in the ViewController.swift file which is used to start the camera session and confirms to the FacialGestureCameraViewDelegate methods.
class ViewController: UIViewController override func viewDidLoad() super. viewDidLoad()// Do any additional setup after loading the view.addCameraViewDelegate()override func viewDidAppear(_ animated: Bool) super.
viewDidAppear(animated)startGestureDetection()override func viewDidDisappear(_ animated: Bool) super. viewDidDisappear(animated)stopGestureDetection()extension ViewController func addCameraViewDelegate() cameraView.delegate = selffunc startGestureDetection() cameraView.
beginSession()func stopGestureDetection() cameraView. stopSession()3. Then we need to create a IBOutLet of FacialGestureCameraView in our view controller.
In order to do that we need to first add a view in ViewControllers Main.storyboard file and assign the class as FacialGestureCameraView as shown below. 4.
Once done create an IBOutlet in ViewController.swift file as shown below.@IBOutlet weak var cameraView: FacialGestureCameraView!
Awesome. We are finally done with the implementation of our delegate methods which will be triggered when a particular face gesture is detected. 5.
Now run the code and check if the delegate methods are getting triggered and if it runs successfully you will see the outputs getting printed in the console.ConclusionIn this article we have made use of the Firebase ML Kit Vision API and have also implemented our custom delegate methods which gets triggered when a particular face gesture is detected. In the Part Three of the series we will learn how to make use of this delegate methods to implement some of the use cases.
The source code for this tutorial can be found here and dont forget to run pod install command before building the project.If you finds these useful, feel free to share this. Thanks for reading!
Till Then RELATED QUESTION Where can I purchase sugar rush kids clothing in wholesale? Hey,Honestly, there are just too many kids wholesale clothing seller in the world, and most of them claim to offer their collection at the cheapest price. And choosing the best wholesale clothing seller is very critical and not easy.
You need kids clothing in wholesale, its good but exactly where are you from, I dont know. If you are from USA, Australia, Canada or Saudi Arabia, I have a suggestion, give a look atAlanic Clothing, one of the bestwholesale kids clothing manufacturer & distributorin the world. They supplies cheap & best wholesale clothing for kids.
And if you needwholesale sublimation kids clothing, I recommend you to check outOasis Sublimation. They have same location as Alanic Clothing, Both they have private label option. I know many people who has bought wholesale clothes from them.
But one thing, you must have your own decision to choose the best wholesale kids clothing seller. Good Luck.
OUR PRODUCT
Brand Ring
Women's Bracelet
Necklace Jewelry
Charming Earrings
CONTACT US
Phone: +86-020-22139325
E-Mail: service@lifisher.com
Add: 5AF Whole Floor, xingguang Yingjing Building, No. 119 Shuiyin Road, Yuexiu District, Guangzhou, China.
Copyright © 2021 Silvergld jewelry-lifisher.com |Sitemap
{"site":{"site_id":2001,"site_type":11,"site_domain":"jewelry-manufacture.com","domain_mode":1,"original_domain":"jewelry-manufacture.com","language_code":"en","is_init_domain":0,"is_shop":true,"is_ssl":1,"lang_prefix":"/","animate_name":"none"},"page":{"page_id":24037,"page_type":"ai_article_detail","page_code":423,"page_url":"/ai-article/firebase-ml-kit-building-a-facial-gesture-detecting-app-in-ios-part-two.html","page_source":"","allowAnimat":0,"content_id":5,"content_type":5,"detail_thumb":"https://img.yfisher.com/1588231418988","detail_title":"Firebase ML Kit: Building A Facial Gesture Detecting App In iOS (Part Two)","moq":1},"translateList":{"A new item has been added to your Shopping Cart":"A new item has been added to your Shopping Cart","account":"account","Account Name":"Account Name","Account Number":"Account Number","Account is not exists":"Account is not exists","account security":"account security","Active Commission":"Active Commission","Add a review on the product":"Add a review on the product","Add to":"Add to","Add to Cart":"add to cart","address book":"address book","affiliate links":"affiliate links","all":"all","All Orders":"All Orders","Already commented":"Already commented","Are you sure to cancel this withdrawal?":"Are you sure to cancel this withdrawal?","Are you sure to delete the selected items?":"Are you sure to delete the selected items?","Are you sure you want to delete it?":"Are you sure you want to delete it?","Awaiting Payment":"Awaiting Payment","Awaiting Shipment":"Awaiting Shipment","Back":"Back","Bank Transfer":"Bank Transfer","bank address":"bank address","basic information":"basic information","Buy":"Buy","Buy Now":"Buy Now","bank name":"bank name","city":"city","Copy successful":"Copy successful","Copy failed":"Copy failed","Can Extract":"Can Extract","Currency Type":"Currency Type","Cancel":"cancel","Cancel the success":"Cancel the success","Cancelled":"Cancelled","Choose a country":"Choose a country","Choose Coupon":"Choose Coupon","Choose items":"Choose items","Clear":"Clear","Clear Search":"Clear Search","Comment Successful!":"Comment Successful!","Comment Failed!":"Comment Failed!","Commission Details":"Commission Details","Commission":"Commission","Commission Status":"Commission Status","commodity payment":"commodity payment","completed":"completed","Completed":"Completed","Condition not met":"Condition not met","Confirm":"Confirm","Confirm password is inconsistent with new password":"confirm password is inconsistent with new password","Congratulations":"Congratulations","Congratulations! You are got a coupon.":"Congratulations! You are got a coupon.","Congratulations! You are got all coupons.":"Congratulations! You are got all coupons.","Continue":"Continue","Continue Shopping":"Continue Shopping","Copy the code and use it directly in the shopping cart.":"Copy the code and use it directly in the shopping cart.","Country":"Country","Coupon code":"Coupon code","Coupon List":"Coupon list","Date":"Date","days after receiving":"days after receiving","Design customization":"Design customization","Do not use any discount":"Do not use any discount","Earliest":"Earliest","Export successful":"Export successful","Export failed":"Export failed","email":"email","email format does not match":"email format does not match","Estimated Delivery Time":"Estimated Delivery Time","Effective Order Count":"Effective Order Count","Effective Sale Amount":"Effective Sale Amount","Expense":"Expense","expired":"expired","export a report?":"export a report?","Failed to upload files.":"Failed to upload files.","FAQ":"FAQ","Find Parts":"Find Parts","for order over":"for order over","Free":"Free","Free Quote & Information Request":"Free Quote & Information Request","Free Shipping":"Free Shipping","Get":"Get","Get coupons":"Get coupons","Get discount":"Get discount","Get it":"Get it","Get it after logging in and use it in the shopping cart.":"Get it after logging in and use it in the shopping cart.","Go to Page":"Go to Page","Highest Price":"Highest Price","home":"home","Hot Sale":"Hot Sale","Income":"Income","Incorrect form format":"Incorrect form format","inquiry":"inquiry","join guide":"join guide","Last 30 days":"Last 30 days","Last 7 days":"Last 7 days","Links report":"Links report","Loading":"Loading","Lowest Price":"Lowest Price","Match Product":"Match Product","Merchant Free Shipping":"Merchant Free Shipping","message":"message","Most Popular":"Most Popular","my account":"my account","my coupons":"my coupons","my inquiry":"my inquiry","my orders":"my orders","my reviews":"my reviews","my wishlist":"my wishlist","name":"name","New Arrival":"New Arrival","Newest":"Newest","No Quotation":"No Quotation","No time limit":"No time limit","Not deleted":"Not deleted","not valid yet":"not valid yet","Off":"Off","Offers and Discounts":"Offers and Discounts","ok":"ok","Only DOC,DOCX,PDF,PNG,JPEG and JPG files can be uploaded":"Only DOC,DOCX,PDF,PNG,JPEG and JPG files can be uploaded","optional":"optional","order notes":"order notes","Order over":"Order over","order id":"order id","order status":"order status","order amount":"order amount","Orders Report":"Orders Report","Other":"Other","Password contains at least numbers and letters length should be 6-20":"password contains at least numbers and letters length should be 6-20","Password is invalid":"Password is invalid","Password length should be 6-20":"Password length should be 6-20","Paypal":"Paypal","paypal payment":"paypal payment","Pending":"Pending","Pending Commission":"Pending Commission","personal info":"personal info","Please click ’click to continue’ to retry.":"Please click \"click to continue\" to retry.","Please contact customer service for cash withdrawal":"Please contact customer service for cash withdrawal","Please enter a valid email address":"Please enter a valid email address","Please enter the verification code":"Please enter the verification code","Please login in first":"Please login in first","Please select attribute":"Please select attribute","Please select country/region":"Please select country/region","Please select superior":"Please select superior","Please select the number of ratings.":"Please select the number of ratings.","Please select your country":"Please select your country","Please upload the invoice file":"Please upload the invoice file","Processing":"Processing","Product Name":"Product Name","Please fill in the delivery address before selecting the payment method":"Please fill in the delivery address before selecting the payment method","promotion center":"promotion center","Promotion Link Click Amount":"Promotion Link Click Amount","Promoted link clicks":"Promoted link clicks","Promotion Order Count":"Promotion Order Count","Promotion Reports":"Promotion Reports","read more":"read more","Received commission":"Received commission","Refund":"Refund","Refuse":"Refuse","Region":"Region","Register Success":"Register Success","Remittance":"Remittance","Reviews":"Reviews","reports":"reports","Sale ends in":"Sale ends in","Save in wishlist":"Save in wishlist","Search":"Search","swift code":"swift code","Select how to share":"Select how to share","Select premium items to increase your chances of making money":"Select premium items to increase your chances of making money","Share items to your channels.when other purchase a from your link, you can get commission.":"Share items to your channels.when other purchase a from your link, you can get commission.","Share Product":"Share Product","shipment successful":"shipment successful","Shipping":"Shipping","Shipping Address":"shipping address","Size guide":"Size guide","Small Text":"Small Text","Small Title":"Small Title","Sort By":"Sort By","Sales Amount":"Sales Amount","State/Province/Territory":"State/Province/Territory","Successfully delete":"Successfully delete","Successfully save":"Successfully save","Thank you for trying":"Thank you for trying","The account has been deactivated, please contact customer service to activate":"The account has been deactivated, please contact customer service to activate","the content can not be blank":"the content can not be blank","The coupon code has been copied and used in the shopping cart.":"The coupon code has been copied and used in the shopping cart.","The file name cannot exceed 100 characters":"The file name cannot exceed 100 characters","The file size cannot exceed 2MB":"The file size cannot exceed 2MB","The number of withdrawals on the day has been capped":"The number of withdrawals on the day has been capped","The subscription is successful, thank you for your participation":"The subscription is successful, thank you for your participation","The user center is out of service. Please contact customer service":"The user center is out of service. Please contact customer service","There is no amount to withdraw":"There is no amount to withdraw","There is no data to export":"There is no data to export","This is Text":"This is Text","This is title":"This is title","This transaction has failed.":"This transaction has failed.","Time to shop":"Time to shop","Tips":"Tips","To be commented":"To be commented","Total":"Total","Tutorial":"Tutorial","This Supplier/Shipping Company does not deliver to your selected Country/Region.":"This Supplier/Shipping Company does not deliver to your selected Country/Region.","Update password success":"Update password success","Upload Image":"Upload Image","Upload up to 6 pictures":"Upload up to 6 pictures","uploading":"uploading","used":"used","user center":"user center","Upload Invoice":"Upload Invoice","valid now":"valid now","Validity period":"Validity period","View Cart & Checkout":"View Cart & Checkout","views":"views","Valid for":"Valid for","Welcome to the website":"Welcome to the website","Western Union":"Western Union","When your buyers received and confirmed orders, you can get commission right now!":"When your buyers received and confirmed orders, you can get commission right now!","Withdrawal":"Withdrawal","Withdrawal success":"Withdrawal success","Withdrawal Method":"Withdrawal Method","Write a Review":"Write a Review","Withdrawal Amount":"Withdrawal Amount","Yes":"Yes","Yesterday":"Yesterday","You are clicking too fast":"You are clicking too fast","You are got a coupon.":"You are got a coupon.","You can select a maximum of 90 days":"You can select a maximum of 90 days","You can withdraw the commission to your Paypal account.":"You can withdraw the commission to your Paypal account.","You haven’t chosen an address yet":"You haven't chosen an address yet","You haven’t selected a product yet":"You haven't selected a product yet","Your rating":"Your rating","Your review":"Your review","Your shipping address error":"Your shipping address error"}}